Human reading behavior is tuned to the statistics of natural language: the
time it takes human subjects to read a word can be predicted from estimates of
the word's probability in context. However, it remains an open question what
computational architecture best characterizes the expectations deployed in real
time by humans that determine the behavioral signatures of reading. Here we
test over two dozen models, independently manipulating computational
architecture and training dataset size, on how well their next-word
expectations predict human reading time behavior on naturalistic text corpora.
We find that across model architectures and training dataset sizes the
relationship between word log-probability and reading time is (near-)linear. We
next evaluate how features of these models determine their psychometric
predictive power, or ability to predict human reading behavior. In general, the
better a model's next-word expectations, the better its psychometric predictive
power. However, we find nontrivial differences across model architectures. For
any given perplexity, deep Transformer models and n-gram models generally show
superior psychometric predictive power over LSTM or structurally supervised
neural models, especially for eye movement data. Finally, we compare models'
psychometric predictive power to the depth of their syntactic knowledge, as
measured by a battery of syntactic generalization tests developed using methods
from controlled psycholinguistic experiments. Once perplexity is controlled
for, we find no significant relationship between syntactic knowledge and
predictive power. These results suggest that different approaches may be
required to best model human real-time language comprehension behavior in
naturalistic reading versus behavior for controlled linguistic materials
designed for targeted probing of syntactic knowledge.

通过对二十多种不同的计算模型测试发现，尽管这些模型可能包含不同的结构、方法和数据集，但其前向单词期望和人类阅读行为之间的关系是很直接的，其中深度 Transformer 模型和 n-gram 模型在眼动数据方面表现优异，而在句法知识和预测能力之间不存在重大关系。

神经语言模型对人类实时理解行为的预测能力

On the Predictive Power of Neural Language Models for Human Real-Time  Comprehension Behavior

When humans read text, they fixate some words and skip others. However, there
have been few attempts to explain skipping behavior with computational models,
as most existing work has focused on predicting reading times (e.g.,~using
surprisal). In this paper, we propose a novel approach that models both
skipping and reading, using an unsupervised architecture that combines a neural
attention with autoencoding, trained on raw text using reinforcement learning.
Our model explains human reading behavior as a tradeoff between precision of
language understanding (encoding the input accurately) and economy of attention
(fixating as few words as possible). We evaluate the model on the Dundee
eye-tracking corpus, showing that it accurately predicts skipping behavior and
reading times, is competitive with surprisal, and captures known qualitative
features of human reading.

本篇论文通过使用无监告运算架构的神经注意力和自编码，探讨了人类阅读时的跳过现象和读取过程的计算，提出了一种新的方法，并在 Dundee 眼球追踪语料库上对该模型进行了评估，表明它能够很好的预测跳过行为和阅读时间，捕捉了人类阅读的已知定性特征。