Two of the central factors believed to underpin human sentence processing
difficulty are expectations and retrieval from working memory. A recent attempt
to create a unified cognitive model integrating these two factors relied on the
parallels between the self-attention mechanism of transformer language models
and cue-based retrieval theories of working memory in human sentence processing
(Ryu and Lewis 2021). While Ryu and Lewis show that attention patterns in
specialized attention heads of GPT-2 are consistent with similarity-based
interference, a key prediction of cue-based retrieval models, their method
requires identifying syntactically specialized attention heads, and makes the
cognitively implausible assumption that hundreds of memory retrieval operations
take place in parallel. In the present work, we develop a recurrent neural
language model with a single self-attention head, which more closely parallels
the memory system assumed by cognitive theories. We show that our model's
single attention head captures semantic and syntactic interference effects
observed in human experiments.

该研究通过开发一个具有单个自注意头的循环神经语言模型，捕捉到了人类实验中观察到的语义和句法干扰效应。