Distributional semantic models capture word-level meaning that is useful in
many natural language processing tasks and have even been shown to capture
cognitive aspects of word meaning. The majority of these models are purely text
based, even though the human sensory experience is much richer. In this paper
we create visually grounded word embeddings by combining English text and
images and compare them to popular text-based methods, to see if visual
information allows our model to better capture cognitive aspects of word
meaning. Our analysis shows that visually grounded embedding similarities are
more predictive of the human reaction times in a large priming experiment than
the purely text-based embeddings. The visually grounded embeddings also
correlate well with human word similarity ratings. Importantly, in both
experiments we show that the grounded embeddings account for a unique portion
of explained variance, even when we include text-based embeddings trained on
huge corpora. This shows that visual grounding allows our model to capture
information that cannot be extracted using text as the only source of
information.

本文旨在利用图像丰富文本语义模型，以更好地捕捉词汇意义的认知方面，并通过大型启动实验展示在视觉基础词嵌入中添加视觉信息的预测性能更好，其中与人类单词相似性的相关性也更高，并展示了视觉基础嵌入可以捕捉到仅使用纯文本不能提取的信息。

看见优势：通过视觉衍生词嵌入更好地捕捉人类的语义知识

Seeing the advantage: visually grounding word embeddings to better capture human semantic knowledge

We present a targeted, scaled-up comparison of incremental processing in
humans and neural language models by collecting by-word reaction time data for
sixteen different syntactic test suites across a range of structural phenomena.
Human reaction time data comes from a novel online experimental paradigm called
the Interpolated Maze task. We compare human reaction times to by-word
probabilities for four contemporary language models, with different
architectures and trained on a range of data set sizes. We find that across
many phenomena, both humans and language models show increased processing
difficulty in ungrammatical sentence regions with human and model `accuracy'
scores (a la Marvin and Linzen(2018)) about equal. However, although language
model outputs match humans in direction, we show that models systematically
under-predict the difference in magnitude of incremental processing difficulty
between grammatical and ungrammatical sentences. Specifically, when models
encounter syntactic violations they fail to accurately predict the longer
reaction times observed in the human data. These results call into question
whether contemporary language models are approaching human-like performance for
sensitivity to syntactic violations.

通过在线实验，比较人类和神经语言模型在语法处理中的情况；发现当神经语言模型遇到语法问题时，会在显著性方面低估人类反应时间的不同，这对于现代语言模型是否接近人类对语法违规的敏感性提出了质疑。