Transformer-based large language models are trained to make predictions about
the next word by aggregating representations of previous tokens through their
self-attention mechanism. In the field of cognitive modeling, such attention
patterns have recently been interpreted as embodying the process of cue-based
retrieval, in which attention over multiple targets is taken to generate
interference and latency during retrieval. Under this framework, this work
first defines an entropy-based predictor that quantifies the diffuseness of
self-attention, as well as distance-based predictors that capture the
incremental change in attention patterns across timesteps. Moreover, following
recent studies that question the informativeness of attention weights, we also
experiment with alternative methods for incorporating vector norms into
attention weights. Regression experiments using predictors calculated from the
GPT-2 language model show that these predictors deliver a substantially better
fit to held-out self-paced reading and eye-tracking data over a rigorous
baseline including GPT-2 surprisal. Additionally, the distance-based predictors
generally demonstrated higher predictive power, with effect sizes of up to 6.59
ms per standard deviation on self-paced reading times (compared to 2.82 ms for
surprisal) and 1.05 ms per standard deviation on eye-gaze durations (compared
to 3.81 ms for surprisal).

本文探讨了 Transformer-based 大型语言模型的自我注意力机制在认知建模领域中的作用，提出了基于熵和距离的预测器，并证明了在使用与 GPT-2 语言模型相关的数据进行回归实验时，距离预测器相较于基于 surprisal 的预测器有更高的预测能力。