Understanding the connections between artificial and biological intelligent
systems can reveal fundamental principles underlying general intelligence.
While many artificial intelligence (AI) models have a neuroscience counterpart,
such connections are largely missing in Transformer models and the
self-attention mechanism. Here, we examine the relationship between attention
heads and human episodic memory. We focus on the induction heads, which
contribute to the in-context learning capabilities of Transformer-based large
language models (LLMs). We demonstrate that induction heads are behaviorally,
functionally, and mechanistically similar to the contextual maintenance and
retrieval (CMR) model of human episodic memory. Our analyses of LLMs
pre-trained on extensive text data show that CMR-like heads often emerge in the
intermediate model layers and that their behavior qualitatively mirrors the
memory biases seen in humans. Our findings uncover a parallel between the
computational mechanisms of LLMs and human memory, offering valuable insights
into both research fields.

通过研究注意力头与人类情节记忆之间的关系，我们发现 Transformers 模型和自注意机制的缺失，并发现在大型语言模型中诱导头的行为、功能和机制与人类情节记忆的上下文维护和检索（CMR）模型有相似之处。我们的研究揭示了 LLMs 的计算机制与人类记忆之间的并行关系，为这两个研究领域提供了有价值的洞见。