A critical challenge to image-text retrieval is how to learn accurate
correspondences between images and texts. Most existing methods mainly focus on
coarse-grained correspondences based on co-occurrences of semantic objects,
while failing to distinguish the fine-grained local correspondences. In this
paper, we propose a novel Scene Graph based Fusion Network (dubbed SGFN), which
enhances the images'/texts' features through intra- and cross-modal fusion for
image-text retrieval. To be specific, we design an intra-modal hierarchical
attention fusion to incorporate semantic contexts, such as objects, attributes,
and relationships, into images'/texts' feature vectors via scene graphs, and a
cross-modal attention fusion to combine the contextual semantics and local
fusion via contextual vectors. Extensive experiments on public datasets
Flickr30K and MSCOCO show that our SGFN performs better than quite a few SOTA
image-text retrieval methods.

本研究提出了一种基于场景图融合网络的图像 - 文本检索方法，通过内部和交叉模态融合来增强图像 / 文本特征，并采用层次化的融合注意力来将语义信息嵌入到图像 / 文本特征向量之中，进而进行图像 - 文本的检索任务。实验结果表明，本方法相比其他现有技术，在公开数据集 Flickr30K 和 MSCOCO 上取得了更好的效果。