Language grounding is an active field aiming at enriching textual representations with visual information. Generally, textual and visual elements are embedded in the same representation space, which implicitly assumes a one-to-one correspondence between modalities. This hypothesis does not hold when representing words, and becomes problematic when used to learn sentence representations --- the focus of this paper --- as a visual scene can be described by a wide variety of sentences. To overcome this limitation, we propose to transfer visual information to textual representations by learning an intermediate representation space: the grounded space. We further propose two new complementary objectives ensuring that (1) sentences associated with the same visual content are close in the grounded space and (2) similarities between related elements are preserved across modalities. We show that this model outperforms the previous state-of-the-art on classification and semantic relatedness tasks.

本文目的在于通过学习一种中间的representation space将视觉信息转移到文本表示中，以解决由于一个视觉场景可以有多种描述方式所导致的文字表示和学习中的问题，作者还提出了两个新的目标，来保证相关元素之间的相似性跨领域地保持和提高分类和语义相关性任务的性能。

在基于图像场景环境的语义空间中融合视觉语义到句子表示中