Image-text matching has received growing interest since it bridges vision and
language. The key challenge lies in how to learn correspondence between image
and text. Existing works learn coarse correspondence based on object
co-occurrence statistics, while failing to learn fine-grained phrase
correspondence. In this paper, we present a novel Graph Structured Matching
Network (GSMN) to learn fine-grained correspondence. The GSMN explicitly models
object, relation and attribute as a structured phrase, which not only allows to
learn correspondence of object, relation and attribute separately, but also
benefits to learn fine-grained correspondence of structured phrase. This is
achieved by node-level matching and structure-level matching. The node-level
matching associates each node with its relevant nodes from another modality,
where the node can be object, relation or attribute. The associated nodes then
jointly infer fine-grained correspondence by fusing neighborhood associations
at structure-level matching. Comprehensive experiments show that GSMN
outperforms state-of-the-art methods on benchmarks, with relative Recall@1
improvements of nearly 7% and 2% on Flickr30K and MSCOCO, respectively. Code
will be released at: this https URL

本文提出了一种新的基于图形结构匹配网络的图像 - 文本匹配方法，可以有效地解决对象、关系和属性的细粒度对应问题。实验结果表明，该方法在 Flickr30K 和 MSCOCO 数据集上相较于其他方法拥有高达近 7% 和 2% 的召回率提升。