Next-word probabilities from language models have been shown to successfully
simulate human reading behavior. Building on this, we show that, interestingly,
instruction-tuned large language models (LLMs) yield worse psychometric
predictive power (PPP) for human reading behavior than base LLMs with
equivalent perplexities. In other words, instruction tuning, which helps LLMs
provide human-preferred responses, does not always make them human-like from
the computational psycholinguistics perspective. In addition, we explore
prompting methodologies in simulating human reading behavior with LLMs, showing
that prompts reflecting a particular linguistic hypothesis lead LLMs to exhibit
better PPP but are still worse than base LLMs. These highlight that recent
instruction tuning and prompting do not offer better estimates than direct
probability measurements from base LLMs in cognitive modeling.

语言模型的下一个词概率已成功模拟了人类阅读行为。然而，在此基础上的研究表明，为了提供人类首选回答而进行的指导调整反而降低了大型语言模型在计算心理语言学视角下的心理测量预测能力。此外，使用特定语言学假设的提示方法仍然不如基础语言模型，暗示最近的指导调整和提示方法在认知建模中无法提供比基础语言模型更好的估计。