Next-word probabilities from language models have been shown to successfully
simulate human reading behavior. Building on this, we show that, interestingly,
instruction-tuned large language models (LLMs) yield worse psychometric
predictive power (PPP) for human reading behavior than base LLMs with
equivalent perplexities. In other words, instruction tuning, which helps LLMs
provide human-preferred responses, does not always make them human-like from
the computational psycholinguistics perspective. In addition, we explore
prompting methodologies in simulating human reading behavior with LLMs, showing
that prompts reflecting a particular linguistic hypothesis lead LLMs to exhibit
better PPP but are still worse than base LLMs. These highlight that recent
instruction tuning and prompting do not offer better estimates than direct
probability measurements from base LLMs in cognitive modeling.

语言模型的下一个词概率已成功模拟了人类阅读行为。然而，在此基础上的研究表明，为了提供人类首选回答而进行的指导调整反而降低了大型语言模型在计算心理语言学视角下的心理测量预测能力。此外，使用特定语言学假设的提示方法仍然不如基础语言模型，暗示最近的指导调整和提示方法在认知建模中无法提供比基础语言模型更好的估计。

大型语言模型的心理测量预测能力

Psychometric Predictive Power of Large Language Models

Human reading behavior is tuned to the statistics of natural language: the
time it takes human subjects to read a word can be predicted from estimates of
the word's probability in context. However, it remains an open question what
computational architecture best characterizes the expectations deployed in real
time by humans that determine the behavioral signatures of reading. Here we
test over two dozen models, independently manipulating computational
architecture and training dataset size, on how well their next-word
expectations predict human reading time behavior on naturalistic text corpora.
We find that across model architectures and training dataset sizes the
relationship between word log-probability and reading time is (near-)linear. We
next evaluate how features of these models determine their psychometric
predictive power, or ability to predict human reading behavior. In general, the
better a model's next-word expectations, the better its psychometric predictive
power. However, we find nontrivial differences across model architectures. For
any given perplexity, deep Transformer models and n-gram models generally show
superior psychometric predictive power over LSTM or structurally supervised
neural models, especially for eye movement data. Finally, we compare models'
psychometric predictive power to the depth of their syntactic knowledge, as
measured by a battery of syntactic generalization tests developed using methods
from controlled psycholinguistic experiments. Once perplexity is controlled
for, we find no significant relationship between syntactic knowledge and
predictive power. These results suggest that different approaches may be
required to best model human real-time language comprehension behavior in
naturalistic reading versus behavior for controlled linguistic materials
designed for targeted probing of syntactic knowledge.

通过对二十多种不同的计算模型测试发现，尽管这些模型可能包含不同的结构、方法和数据集，但其前向单词期望和人类阅读行为之间的关系是很直接的，其中深度 Transformer 模型和 n-gram 模型在眼动数据方面表现优异，而在句法知识和预测能力之间不存在重大关系。