In this paper we propose to learn a multimodal image and text embedding from
Web and Social Media data, aiming to leverage the semantic knowledge learnt in
the text domain and transfer it to a visual model for semantic image retrieval.
We demonstrate that the pipeline can learn from images with associated text
without supervision and perform a thourough analysis of five different text
embeddings in three different benchmarks. We show that the embeddings learnt
with Web and Social Media data have competitive performances over supervised
methods in the text based image retrieval task, and we clearly outperform state
of the art in the MIRFlickr dataset when training in the target data. Further
we demonstrate how semantic multimodal image retrieval can be performed using
the learnt embeddings, going beyond classical instance-level retrieval
problems. Finally, we present a new dataset, InstaCities1M, composed by
Instagram images and their associated texts that can be used for fair
comparison of image-text embeddings.

本研究提出利用网络和社交媒体数据来学习多模态图像和文本嵌入，旨在将在文本领域中学到的语义知识转移至用于语义图像检索的视觉模型。研究结果表明，利用带有相关文本的图像进行无监督学习的流程能够在三个基准测试中学习五种不同的文本嵌入，并在面向文本的图像检索任务中表现出与受监督方法竞争性的性能。在目标数据中训练时，我们在 MIRFlickr 数据集中明显优于现有技术。进一步，我们展示了如何使用学习到的嵌入执行语义多模态图像检索，超越了传统的实例级检索问题。最后，我们提出了一个新的数据集（InstaCities1M），由 Instagram 图像及其相关文本组成，可用于公平比较图像 - 文本嵌入方法。

从网络数据中通过深度语义嵌入学习学习

Learning to Learn from Web Data through Deep Semantic Embeddings

We address personalization issues of image captioning, which have not been
discussed yet in previous research. For a query image, we aim to generate a
descriptive sentence, accounting for prior knowledge such as the user's active
vocabularies in previous documents. As applications of personalized image
captioning, we tackle two post automation tasks: hashtag prediction and post
generation, on our newly collected Instagram dataset, consisting of 1.1M posts
from 6.3K users. We propose a novel captioning model named Context Sequence
Memory Network (CSMN). Its unique updates over previous memory network models
include (i) exploiting memory as a repository for multiple types of context
information, (ii) appending previously generated words into memory to capture
long-term information without suffering from the vanishing gradient problem,
and (iii) adopting CNN memory structure to jointly represent nearby ordered
memory slots for better context understanding. With quantitative evaluation and
user studies via Amazon Mechanical Turk, we show the effectiveness of the three
novel features of CSMN and its performance enhancement for personalized image
captioning over state-of-the-art captioning models.

本文探讨了图像字幕的个性化问题，提出了一种名为 Context Sequence Memory Network（CSMN）的新型字幕模型，结合先前生成的单词的记忆信息以及有关个人先前文档中使用的词汇的前置知识生成对查询图像的描述性句子，并在新收集的 Instagram 数据集上进行应用，展示了其对状态 - of-the-art 字幕模型的性能改进.