We examine the benefits of visual context in training neural language models
to perform next-word prediction. A multi-modal neural architecture is
introduced that outperform its equivalent trained on language alone with a 2\%
decrease in perplexity, even when no visual context is available at test.
Fine-tuning the embeddings of a pre-trained state-of-the-art bidirectional
language model (BERT) in the language modeling framework yields a 3.5\%
improvement. The advantage for training with visual context when testing
without is robust across different languages (English, German and Spanish) and
different models (GRU, LSTM, $\Delta$-RNN, as well as those that use BERT
embeddings). Thus, language models perform better when they learn like a baby,
i.e, in a multi-modal environment. This finding is compatible with the theory
of situated cognition: language is inseparable from its physical context.

通过实验证明了在多模态环境中进行语言学习可以提高预测准确率，该研究使用了预训练的 BERT 嵌入以及不同语言和模型进行了测试，并得出了这个认识与身处环境相应认知理论相对应的结论。