Self-Supervised learning from multimodal image and text data allows deep neural networks to learn powerful features with no need of human annotated data. Web and Social Media platforms provide a virtually unlimited amount of this multimodal data. In this work we propose to exploit this free available data to learn a multimodal image and text embedding, aiming to leverage the semantic knowledge learnt in the text domain and transfer it to a visual model for semantic image retrieval. We demonstrate that the proposed pipeline can learn from images with associated textwithout supervision and analyze the semantic structure of the learnt joint image and text embedding space. We perform a thorough analysis and performance comparison of five different state of the art text embeddings in three different benchmarks. We show that the embeddings learnt with Web and Social Media data have competitive performances over supervised methods in the text based image retrieval task, and we clearly outperform state of the art in the MIRFlickr dataset when training in the target data. Further, we demonstrate how semantic multimodal image retrieval can be performed using the learnt embeddings, going beyond classical instance-level retrieval problems. Finally, we present a new dataset, InstaCities1M, composed by Instagram images and their associated texts that can be used for fair comparison of image-text embeddings.

通过利用 Web 和 Social Media 数据，本文提出一种利用多模态图像和文本嵌入的自监督学习方法，在不需要人工注释的情况下学习强大的特征，并将文本领域学到的语义知识转移至视觉模型用于语义图像检索任务。研究分析了五种不同的文本嵌入方法，表明利用 Web 和 Social Media 数据学习的嵌入具有与监督方法相当的性能，且在训练目标数据时优于最先进方法。最后，介绍了 InstaCities1M 数据集，并演示了如何利用该数据集进行语义多模态图像检索。

基于网络数据的自监督学习在多模态检索中的应用