Recent psycholinguistic studies have drawn conflicting conclusions about the
relationship between the quality of a language model and the ability of its
surprisal estimates to predict human reading times, which has been speculated
to be due to the large gap in both the amount of training data and model
capacity across studies. The current work aims to consolidate these findings by
evaluating surprisal estimates from Transformer-based language model variants
that vary systematically in the amount of training data and model capacity on
their ability to predict human reading times. The results show that surprisal
estimates from most variants with contemporary model capacities provide the
best fit after seeing about two billion training tokens, after which they begin
to diverge from humanlike expectations. Additionally, newly-trained smaller
model variants reveal a 'tipping point' at convergence, after which the
decrease in language model perplexity begins to result in poorer fits to human
reading times. These results suggest that the massive amount of training data
is mainly responsible for the poorer fit achieved by surprisal from larger
pre-trained language models, and that a certain degree of model capacity is
necessary for Transformer-based language models to capture humanlike
expectations.

本文研究了基于 Transformer 的语言模型中，各种训练数据和不同容量的模型对于预测人类阅读时间的作用，并发现多数具有当代模型能力的变体，使用约 20 亿个训练标记后，所给出的 surprisal estimates 提供了最佳适合度，而较大的预先训练语言模型的较差适合度主要归咎于大量的训练数据，而 transformer-based 语言模型的某种程度的模型容量对于模型要捕捉类似于人类的期望是必要的。

基于 Transformer 的语言模型惊奇度在使用约 20 亿训练令牌时最能预测人类阅读时间

Transformer-Based LM Surprisal Predicts Human Reading Times Best with  About Two Billion Training Tokens

It has been argued that humans rapidly adapt their lexical and syntactic
expectations to match the statistics of the current linguistic context. We
provide further support to this claim by showing that the addition of a simple
adaptation mechanism to a neural language model improves our predictions of
human reading times compared to a non-adaptive model. We analyze the
performance of the model on controlled materials from psycholinguistic
experiments and show that it adapts not only to lexical items but also to
abstract syntactic structures.

研究表明，人类能够快速适应当前语言环境的词汇和句法预期，添加一个简单的适应机制到神经语言模型中可以提高我们对人类阅读时间的预测准确度，该模型不仅可以适应词汇，还可以适应抽象的句法结构。