Cross-modal information retrieval aims to find heterogeneous data of various
modalities from a given query of one modality. The main challenge is to map
different modalities into a common semantic space, in which distance between
concepts in different modalities can be well modeled. For cross-modal
information retrieval between images and texts, existing work mostly uses
off-the-shelf Convolutional Neural Network (CNN) for image feature extraction.
For texts, word-level features such as bag-of-words or word2vec are employed to
build deep learning models to represent texts. Besides word-level semantics,
the semantic relations between words are also informative but less explored. In
this paper, we model texts by graphs using similarity measure based on
word2vec. A dual-path neural network model is proposed for couple feature
learning in cross-modal information retrieval. One path utilizes Graph
Convolutional Network (GCN) for text modeling based on graph representations.
The other path uses a neural network with layers of nonlinearities for image
modeling based on off-the-shelf features. The model is trained by a pairwise
similarity loss function to maximize the similarity of relevant text-image
pairs and minimize the similarity of irrelevant pairs. Experimental results
show that the proposed model outperforms the state-of-the-art methods
significantly, with 17% improvement on accuracy for the best case.

本研究提出了一种基于图的文本建模方法和双向神经网络模型，用于跨模态信息检索，结合卷积神经网络和 Graph Convolutional Network，通过相似度度量，并采用成对相似度损失函数训练模型，在准确率提升了 17％的前提下，成为了最先进的方法。