Recent models for cross-modal retrieval have benefited from an increasingly
rich understanding of visual scenes, afforded by scene graphs and object
interactions to mention a few. This has resulted in an improved matching
between the visual representation of an image and the textual representation of
its caption. Yet, current visual representations overlook a key aspect: the
text appearing in images, which may contain crucial information for retrieval.
In this paper, we first propose a new dataset that allows exploration of
cross-modal retrieval where images contain scene-text instances. Then, armed
with this dataset, we describe several approaches which leverage scene text,
including a better scene-text aware cross-modal retrieval method which uses
specialized representations for text from the captions and text from the visual
scene, and reconcile them in a common embedding space. Extensive experiments
confirm that cross-modal retrieval approaches benefit from scene text and
highlight interesting research questions worth exploring further. Dataset and
code are available at this http URL

本文提出了一个新的数据集，使得可以探索图像包含场景文本实例时的交叉检索。我们提出了几种方法，其中包括更好的场景文本感知交叉检索方法，它使用了来自标题和视觉场景文本的专门表示，并将它们调和在一个公共嵌入空间中。大量实验证实了这些方法从场景文本中受益，并突出了值得进一步探索的有趣研究问题。本文中提出的数据集和代码可在 http URL 中获得。

StacMR: 场景文本感知的跨模态检索

StacMR: Scene-Text Aware Cross-Modal Retrieval

Maybe the single most important goal of representation learning is making
subsequent learning faster. Surprisingly, this fact is not well reflected in
the way embeddings are evaluated. In addition, recent practice in word
embeddings points towards importance of learning specialized representations.
We argue that focus of word representation evaluation should reflect those
trends and shift towards evaluating what useful information is easily
accessible. Specifically, we propose that evaluation should focus on data
efficiency and simple supervised tasks, where the amount of available data is
varied and scores of a supervised model are reported for each subset (as
commonly done in transfer learning).
In order to illustrate significance of such analysis, a comprehensive
evaluation of selected word embeddings is presented. Proposed approach yields a
more complete picture and brings new insight into performance characteristics,
for instance information about word similarity or analogy tends to be
non--linearly encoded in the embedding space, which questions the cosine-based,
unsupervised, evaluation methods. All results and analysis scripts are
available online.

提出了一种基于数据效率和简单监督任务的评估方法，系统评估了一些词嵌入模型，并得出了新的性能特征见解，例如词相似性和类比常常用非线性编码，质疑了基于余弦相似度的无监督评估方法。