The visual explanation of learned representation of models helps to
understand the fundamentals of learning. The attentional models of previous
works used to visualize the attended regions over an image or text using their
learned weights to confirm their intended mechanism. Kim et al. (2016) show
that the Hadamard product in multimodal deep networks, which is well-known for
the joint function of visual question answering tasks, implicitly performs an
attentional mechanism for visual inputs. In this work, we extend their work to
show that the Hadamard product in multimodal deep networks performs not only
for visual inputs but also for textual inputs simultaneously using the proposed
gradient-based visualization technique. The attentional effect of Hadamard
product is visualized for both visual and textual inputs by analyzing the two
inputs and an output of the Hadamard product with the proposed method and
compared with learned attentional weights of a visual question answering model.

本文拓展了 Kim et al. (2016) 的工作，提出了一种基于梯度的可视化技术，证明了多模式深度网络中的 Hadamard 乘积不仅适用于视觉输入，同时适用于文本输入，并可通过该技术可视化 Hadamard 乘积对视觉和文本输入的注意力机制。

多模深度网络中 Hadamard 乘积的可视化解释

Visual Explanations from Hadamard Product in Multimodal Deep Networks

Deep neural networks continue to advance the state-of-the-art of image
recognition tasks with various methods. However, applications of these methods
to multimodality remain limited. We present Multimodal Residual Networks (MRN)
for the multimodal residual learning of visual question-answering, which
extends the idea of the deep residual learning. Unlike the deep residual
learning, MRN effectively learns the joint representation from vision and
language information. The main idea is to use element-wise multiplication for
the joint residual mappings exploiting the residual learning of the attentional
models in recent studies. Various alternative models introduced by
multimodality are explored based on our study. We achieve the state-of-the-art
results on the Visual QA dataset for both Open-Ended and Multiple-Choice tasks.
Moreover, we introduce a novel method to visualize the attention effect of the
joint representations for each learning block using back-propagation algorithm,
even though the visual features are collapsed without spatial information.

介绍了 Multimodal Residual Networks（MRN），一种通过联合残差映射学习视觉问答的多模态系统，利用注意力模型的残差学习来将视觉和语言信息联合表示，获得了 Visual QA 数据集开放式和多项选择任务的最先进结果，并通过反向传播算法引入了一种可视化联合表示的注意力效果的新方法。

多模态残差学习用于视觉问答

Multimodal Residual Learning for Visual QA

In many computer vision tasks, the relevant information to solve the problem
at hand is mixed to irrelevant, distracting information. This has motivated
researchers to design attentional models that can dynamically focus on parts of
images or videos that are salient, e.g., by down-weighting irrelevant pixels.
In this work, we propose a spatiotemporal attentional model that learns where
to look in a video directly from human fixation data. We model visual attention
with a mixture of Gaussians at each frame. This distribution is used to express
the probability of saliency for each pixel. Time consistency in videos is
modeled hierarchically by: 1) deep 3D convolutional features to represent
spatial and short-term time relations and 2) a long short-term memory network
on top that aggregates the clip-level representation of sequential clips and
therefore expands the temporal domain from few frames to seconds. The
parameters of the proposed model are optimized via maximum likelihood
estimation using human fixations as training data, without knowledge of the
action in each video. Our experiments on Hollywood2 show state-of-the-art
performance on saliency prediction for video. We also show that our attentional
model trained on Hollywood2 generalizes well to UCF101 and it can be leveraged
to improve action classification accuracy on both datasets.

本研究提出了一种基于高斯混合模型的时空注意力模型，并使用人类注视数据进行训练，相比先前的方法，该模型在影片显著性预测方面具有最先进的表现，并在动作分类准确性方面取得了改进。