Image and sentence matching has made great progress recently, but it remains
challenging due to the large visual-semantic discrepancy. This mainly arises
from that the representation of pixel-level image usually lacks of high-level
semantic information as in its matched sentence. In this work, we propose a
semantic-enhanced image and sentence matching model, which can improve the
image representation by learning semantic concepts and then organizing them in
a correct semantic order. Given an image, we first use a multi-regional
multi-label CNN to predict its semantic concepts, including objects,
properties, actions, etc. Then, considering that different orders of semantic
concepts lead to diverse semantic meanings, we use a context-gated sentence
generation scheme for semantic order learning. It simultaneously uses the image
global context containing concept relations as reference and the groundtruth
semantic order in the matched sentence as supervision. After obtaining the
improved image representation, we learn the sentence representation with a
conventional LSTM, and then jointly perform image and sentence matching and
sentence generation for model learning. Extensive experiments demonstrate the
effectiveness of our learned semantic concepts and order, by achieving the
state-of-the-art results on two public benchmark datasets.

该研究提出了一种基于学习语义概念并按照正确语义顺序组织它们来提高图像表示的语义增强图像和句子匹配模型，利用多区域多标签 CNN 预测图像语义概念并使用上下文门控句子生成方案进行语义顺序学习，与传统 LSTM 结合执行图像和句子匹配和生成。这项工作在两个公共基准数据集上实现了最先进的结果。