We present a targeted, scaled-up comparison of incremental processing in
humans and neural language models by collecting by-word reaction time data for
sixteen different syntactic test suites across a range of structural phenomena.
Human reaction time data comes from a novel online experimental paradigm called
the Interpolated Maze task. We compare human reaction times to by-word
probabilities for four contemporary language models, with different
architectures and trained on a range of data set sizes. We find that across
many phenomena, both humans and language models show increased processing
difficulty in ungrammatical sentence regions with human and model `accuracy'
scores (a la Marvin and Linzen(2018)) about equal. However, although language
model outputs match humans in direction, we show that models systematically
under-predict the difference in magnitude of incremental processing difficulty
between grammatical and ungrammatical sentences. Specifically, when models
encounter syntactic violations they fail to accurately predict the longer
reaction times observed in the human data. These results call into question
whether contemporary language models are approaching human-like performance for
sensitivity to syntactic violations.

通过在线实验，比较人类和神经语言模型在语法处理中的情况；发现当神经语言模型遇到语法问题时，会在显著性方面低估人类反应时间的不同，这对于现代语言模型是否接近人类对语法违规的敏感性提出了质疑。