Several works have proposed to learn a two-path neural network that maps
images and texts, respectively, to a same shared Euclidean space where geometry
captures useful semantic relationships. Such a multi-modal embedding can be
trained and used for various tasks, notably image captioning. In the present
work, we introduce a new architecture of this type, with a visual path that
leverages recent space-aware pooling mechanisms. Combined with a textual path
which is jointly trained from scratch, our semantic-visual embedding offers a
versatile model. Once trained under the supervision of captioned images, it
yields new state-of-the-art performance on cross-modal retrieval. It also
allows the localization of new concepts from the embedding space into any input
image, delivering state-of-the-art result on the visual grounding of phrases.

本研究提出了一种新的两条路径的神经网络，其中视觉路径采用了最新的空间感知池化机制模型，结合从头开始训练的文本路径，实现了一种多模态嵌入。在处理带有标注图像的任务中经过训练后，该模型可提供新的跨模态检索性能和短语的视觉定位，达到了最新的最佳表现。