Existing image-text matching approaches typically infer the similarity of an image-text pair by capturing and aggregating the affinities between the text and each independent object of the image. However, they ignore the connections between the objects that are semantically related. These objects may collectively determine whether the image corresponds to a text or not. To address this problem, we propose a Dual Path Recurrent Neural Network (DP-RNN) which processes images and sentences symmetrically by recurrent neural networks (RNN). In particular, given an input image-text pair, our model reorders the image objects based on the positions of their most related words in the text. In the same way as extracting the hidden features from word embeddings, the model leverages RNN to extract high-level object features from the reordered object inputs. We validate that the high-level object features contain useful joint information of semantically related objects, which benefit the retrieval task. To compute the image-text similarity, we incorporate a Multi-attention Cross Matching Model into DP-RNN. It aggregates the affinity between objects and words with cross-modality guided attention and self-attention. Our model achieves the state-of-the-art performance on Flickr30K dataset and competitive performance on MS-COCO dataset. Extensive experiments demonstrate the effectiveness of our model.

文章提出了一种针对图像-文本匹配问题的解决方法，利用一种双通路递归神经网络（DP-RNN）处理对称的输入，通过提取对象顺序信息、对象关系、同时引入自注意力和跨模态联合注意力实现了图片和文本的相似度匹配，实验验证提出的方法在Flickr30K数据集上达到了最先进的性能表现，MS-COCO数据集也表现具有竞争力。

像语言一样表达物体: 用于图像-文本匹配的循环视觉嵌入