This paper proposes a cross-modal retrieval system that leverages on image
and text encoding. Most multimodal architectures employ separate networks for
each modality to capture the semantic relationship between them. However, in
our work image-text encoding can achieve comparable results in terms of
cross-modal retrieval without having to use a separate network for each
modality. We show that text encodings can capture semantic relationships
between multiple modalities. In our knowledge, this work is the first of its
kind in terms of employing a single network and fused image-text embedding for
cross-modal retrieval. We evaluate our approach on two famous multimodal
datasets: MS-COCO and Flickr30K.

本文提出了一种交叉模态检索系统，利用图像和文本编码，实现了同时检索模态的功能，避免了需要为每个模态使用不同网络的缺点。在所使用的知识中，本文是第一篇采用单一网络和融合的图像 - 文本嵌入进行跨模态检索的工作。在 MS-COCO 和 Flickr30K 两个著名的多模态数据集上对该方法进行了评估。

重新审视跨模态检索

Revisiting Cross Modal Retrieval

In this paper, we study the problem of image-text matching. Inferring the
latent semantic alignment between objects or other salient stuff (e.g. snow,
sky, lawn) and the corresponding words in sentences allows to capture
fine-grained interplay between vision and language, and makes image-text
matching more interpretable. Prior work either simply aggregates the similarity
of all possible pairs of regions and words without attending differentially to
more and less important words or regions, or uses a multi-step attentional
process to capture limited number of semantic alignments which is less
interpretable. In this paper, we present Stacked Cross Attention to discover
the full latent alignments using both image regions and words in a sentence as
context and infer image-text similarity. Our approach achieves the
state-of-the-art results on the MS-COCO and Flickr30K datasets. On Flickr30K,
our approach outperforms the current best methods by 22.1% relatively in text
retrieval from image query, and 18.2% relatively in image retrieval with text
query (based on Recall@1). On MS-COCO, our approach improves sentence retrieval
by 17.8% relatively and image retrieval by 16.6% relatively (based on Recall@1
using the 5K test set). Code has been made available at:
this https URL

本文研究图像文本匹配问题，提出了一种使用交叉注意力机制的方法来发现包括图像区域和句子单词在内的全部潜在对齐，从而推断图像与文本之间的相似性，实验结果在 MS-COCO 和 Flickr30K 数据集上均表现为最先进的水平。