We introduce a model for bidirectional retrieval of images and sentences
through a multi-modal embedding of visual and natural language data. Unlike
previous models that directly map images or sentences into a common embedding
space, our model works on a finer level and embeds fragments of images
(objects) and fragments of sentences (typed dependency tree relations) into a
common space. In addition to a ranking objective seen in previous work, this
allows us to add a new fragment alignment objective that learns to directly
associate these fragments across modalities. Extensive experimental evaluation
shows that reasoning on both the global level of images and sentences and the
finer level of their respective fragments significantly improves performance on
image-sentence retrieval tasks. Additionally, our model provides interpretable
predictions since the inferred inter-modal fragment alignment is explicit.

该研究提出了一种通过多模态的可视图像和自然语言数据的嵌入来实现图像和句子的双向检索的模型，通过对图像和句子的片段进行嵌入到一个共同的空间，实现了片段对齐，从而在图像句子检索任务中显著提高了性能，并提供了可解释的预测结果。