Language grounding to vision is an active field of research aiming to enrich
text-based representations of word meanings by leveraging perceptual knowledge
from vision. Despite many attempts at language grounding, it is still unclear
how to effectively inject visual knowledge into the word embeddings of a
language in such a way that a proper balance of textual and visual knowledge is
maintained. Some common concerns are the following. Is visual grounding
beneficial for abstract words or is its contribution only limited to concrete
words? What is the optimal way of bridging the gap between text and vision? How
much do we gain by visually grounding textual embeddings? The present study
addresses these questions by proposing a simple yet very effective grounding
approach for pre-trained word embeddings. Our model aligns textual embeddings
with vision while largely preserving the distributional statistics that
characterize word use in text corpora. By applying a learned alignment, we are
able to generate visually grounded embeddings for unseen words, including
abstract words. A series of evaluations on word similarity benchmarks shows
that visual grounding is beneficial not only for concrete words, but also for
abstract words. We also show that our method for visual grounding offers
advantages for contextualized embeddings, but only when these are trained on
corpora of relatively modest size. Code and grounded embeddings for English are
available at this https URL.

本研究提出了一种简单且非常有效的预训练词嵌入视觉对齐方法，使得即使对于抽象词，我们也能生成受视觉基础支撑的嵌入，并且在一系列的单词相似性基准测试中证明了视觉基础支撑不仅有益于具体词，还有益于抽象词。