Distributional semantic models capture word-level meaning that is useful in
many natural language processing tasks and have even been shown to capture
cognitive aspects of word meaning. The majority of these models are purely text
based, even though the human sensory experience is much richer. In this paper
we create visually grounded word embeddings by combining English text and
images and compare them to popular text-based methods, to see if visual
information allows our model to better capture cognitive aspects of word
meaning. Our analysis shows that visually grounded embedding similarities are
more predictive of the human reaction times in a large priming experiment than
the purely text-based embeddings. The visually grounded embeddings also
correlate well with human word similarity ratings. Importantly, in both
experiments we show that the grounded embeddings account for a unique portion
of explained variance, even when we include text-based embeddings trained on
huge corpora. This shows that visual grounding allows our model to capture
information that cannot be extracted using text as the only source of
information.

本文旨在利用图像丰富文本语义模型，以更好地捕捉词汇意义的认知方面，并通过大型启动实验展示在视觉基础词嵌入中添加视觉信息的预测性能更好，其中与人类单词相似性的相关性也更高，并展示了视觉基础嵌入可以捕捉到仅使用纯文本不能提取的信息。

看见优势：通过视觉衍生词嵌入更好地捕捉人类的语义知识

Seeing the advantage: visually grounding word embeddings to better capture human semantic knowledge

We propose a model to learn visually grounded word embeddings (vis-w2v) to
capture visual notions of semantic relatedness. While word embeddings trained
using text have been extremely successful, they cannot uncover notions of
semantic relatedness implicit in our visual world. For instance, although
"eats" and "stares at" seem unrelated in text, they share semantics visually.
When people are eating something, they also tend to stare at the food.
Grounding diverse relations like "eats" and "stares at" into vision remains
challenging, despite recent progress in vision. We note that the visual
grounding of words depends on semantics, and not the literal pixels. We thus
use abstract scenes created from clipart to provide the visual grounding. We
find that the embeddings we learn capture fine-grained, visually grounded
notions of semantic relatedness. We show improvements over text-only word
embeddings (word2vec) on three tasks: common-sense assertion classification,
visual paraphrasing and text-based image retrieval. Our code and datasets are
available online.

本文提出了一种能够捕捉语义相似性的视觉词向量嵌入模型，将抽象场景作为视觉基础，并在三项任务中取得了超越文本词向量的表现。