Unlike most neural language models, humans learn language in a rich,
multi-sensory and, often, multi-lingual environment. Current language models
typically fail to fully capture the complexities of multilingual language use.
We train an LSTM language model on images and captions in English and Spanish
from MS-COCO-ES. We find that the visual grounding improves the model's
understanding of semantic similarity both within and across languages and
improves perplexity. However, we find no significant advantage of visual
grounding for abstract words. Our results provide additional evidence of the
advantages of visually grounded language models and point to the need for more
naturalistic language data from multilingual speakers and multilingual datasets
with perceptual grounding.

通过使用 MS-COCO-ES 数据集中的英语和西班牙语的图像和字幕来训练 LSTM 语言模型，研究发现视觉基础能够提高语义相似性的理解能力，特别是在跨语言时，但是在抽象词汇方面没有显著优势。研究还指出，为了进一步提高视觉基础的语言模型的实用性，需要更多多语言数据和多语言说话者的感知基础。