In this paper we propose to learn a multimodal image and text embedding from
Web and Social Media data, aiming to leverage the semantic knowledge learnt in
the text domain and transfer it to a visual model for semantic image retrieval.
We demonstrate that the pipeline can learn from images with associated text
without supervision and perform a thourough analysis of five different text
embeddings in three different benchmarks. We show that the embeddings learnt
with Web and Social Media data have competitive performances over supervised
methods in the text based image retrieval task, and we clearly outperform state
of the art in the MIRFlickr dataset when training in the target data. Further
we demonstrate how semantic multimodal image retrieval can be performed using
the learnt embeddings, going beyond classical instance-level retrieval
problems. Finally, we present a new dataset, InstaCities1M, composed by
Instagram images and their associated texts that can be used for fair
comparison of image-text embeddings.

本研究提出利用网络和社交媒体数据来学习多模态图像和文本嵌入，旨在将在文本领域中学到的语义知识转移至用于语义图像检索的视觉模型。研究结果表明，利用带有相关文本的图像进行无监督学习的流程能够在三个基准测试中学习五种不同的文本嵌入，并在面向文本的图像检索任务中表现出与受监督方法竞争性的性能。在目标数据中训练时，我们在 MIRFlickr 数据集中明显优于现有技术。进一步，我们展示了如何使用学习到的嵌入执行语义多模态图像检索，超越了传统的实例级检索问题。最后，我们提出了一个新的数据集（InstaCities1M），由 Instagram 图像及其相关文本组成，可用于公平比较图像 - 文本嵌入方法。