Image-language matching tasks have recently attracted a lot of attention in
the computer vision field. These tasks include image-sentence matching, i.e.,
given an image query, retrieving relevant sentences and vice versa, and
region-phrase matching or visual grounding, i.e., matching a phrase to relevant
regions. This paper investigates two-branch neural networks for learning the
similarity between these two data modalities. We propose two network structures
that produce different output representations. The first one, referred to as an
embedding network, learns an explicit shared latent embedding space with a
maximum-margin ranking loss and novel neighborhood constraints. Compared to
standard triplet sampling, we perform improved neighborhood sampling that takes
neighborhood information into consideration while constructing mini-batches.
The second network structure, referred to as a similarity network, fuses the
two branches via element-wise product and is trained with regression loss to
directly predict a similarity score. Extensive experiments show that our
networks achieve high accuracies for phrase localization on the Flickr30K
Entities dataset and for bi-directional image-sentence retrieval on Flickr30K
and MSCOCO datasets.

本文研究了两种神经网络结构用于学习图像和语言之间的相似性，其中一种网络结构学习了一个共享的潜在嵌入空间，另一种网络结构通过元素乘积将两个数据模态融合，并且实现了对于句子和图像的双向检索，取得了良好的结果。