In this paper, we study the problem of image-text matching. Inferring the
latent semantic alignment between objects or other salient stuff (e.g. snow,
sky, lawn) and the corresponding words in sentences allows to capture
fine-grained interplay between vision and language, and makes image-text
matching more interpretable. Prior work either simply aggregates the similarity
of all possible pairs of regions and words without attending differentially to
more and less important words or regions, or uses a multi-step attentional
process to capture limited number of semantic alignments which is less
interpretable. In this paper, we present Stacked Cross Attention to discover
the full latent alignments using both image regions and words in a sentence as
context and infer image-text similarity. Our approach achieves the
state-of-the-art results on the MS-COCO and Flickr30K datasets. On Flickr30K,
our approach outperforms the current best methods by 22.1% relatively in text
retrieval from image query, and 18.2% relatively in image retrieval with text
query (based on Recall@1). On MS-COCO, our approach improves sentence retrieval
by 17.8% relatively and image retrieval by 16.6% relatively (based on Recall@1
using the 5K test set). Code has been made available at:
this https URL

本文研究图像文本匹配问题，提出了一种使用交叉注意力机制的方法来发现包括图像区域和句子单词在内的全部潜在对齐，从而推断图像与文本之间的相似性，实验结果在 MS-COCO 和 Flickr30K 数据集上均表现为最先进的水平。

图像文本匹配的交叉堆叠注意力

Stacked Cross Attention for Image-Text Matching

Matching images and sentences demands a fine understanding of both
modalities. In this paper, we propose a new system to discriminatively embed
the image and text to a shared visual-textual space. In this field, most
existing works apply the ranking loss to pull the positive image / text pairs
close and push the negative pairs apart from each other. However, directly
deploying the ranking loss is hard for network learning, since it starts from
the two heterogeneous features to build inter-modal relationship. To address
this problem, we propose the instance loss which explicitly considers the
intra-modal data distribution. It is based on an unsupervised assumption that
each image / text group can be viewed as a class. So the network can learn the
fine granularity from every image/text group. The experiment shows that the
instance loss offers better weight initialization for the ranking loss, so that
more discriminative embeddings can be learned. Besides, existing works usually
apply the off-the-shelf features, i.e., word2vec and fixed visual feature. So
in a minor contribution, this paper constructs an end-to-end dual-path
convolutional network to learn the image and text representations. End-to-end
learning allows the system to directly learn from the data and fully utilize
the supervision. On two generic retrieval datasets (Flickr30k and MSCOCO),
experiments demonstrate that our method yields competitive accuracy compared to
state-of-the-art methods. Moreover, in language based person retrieval, we
improve the state of the art by a large margin. The code has been made publicly
available.

本文提出了一种新的系统，以区分性地嵌入图像和文本到共享的视觉 - 文本空间，通过实例损失和端到端学习的双路径卷积网络来解决图像和文本匹配问题，并在 Flickr 30k 和 MSCOCO 上达到了与最先进方法相当的准确度，同时在基于语言的人员检索中取得了很大的改进。

带有实例损失的双路径卷积图像 - 文本嵌入

Dual-Path Convolutional Image-Text Embeddings with Instance Loss

We propose Dual Attention Networks (DANs) which jointly leverage visual and
textual attention mechanisms to capture fine-grained interplay between vision
and language. DANs attend to specific regions in images and words in text
through multiple steps and gather essential information from both modalities.
Based on this framework, we introduce two types of DANs for multimodal
reasoning and matching, respectively. The reasoning model allows visual and
textual attentions to steer each other during collaborative inference, which is
useful for tasks such as Visual Question Answering (VQA). In addition, the
matching model exploits the two attention mechanisms to estimate the similarity
between images and sentences by focusing on their shared semantics. Our
extensive experiments validate the effectiveness of DANs in combining vision
and language, achieving the state-of-the-art performance on public benchmarks
for VQA and image-text matching.

提出双重关注网络框架（Dual Attention Networks），应用于视觉和文本场景的多模态推理、比对任务，在 Visual Question Answering 和图片 - 文本比对任务中取得最新的优异表现。