Large Language Models (LLMs) have the capacity to store and recall facts.
Through experimentation with open-source models, we observe that this ability
to retrieve facts can be easily manipulated by changing contexts, even without
altering their factual meanings. These findings highlight that LLMs might
behave like an associative memory model where certain tokens in the contexts
serve as clues to retrieving facts. We mathematically explore this property by
studying how transformers, the building blocks of LLMs, can complete such
memory tasks. We study a simple latent concept association problem with a
one-layer transformer and we show theoretically and empirically that the
transformer gathers information using self-attention and uses the value matrix
for associative memory.

大型语言模型具有存储和提取事实的能力，并且可以通过改变上下文来操纵提取事实的能力，揭示出它们可能像联想记忆模型一样行为，其中上下文中的某些令牌作为提取事实的线索。我们通过研究 transformer 如何完成此类记忆任务，对这一属性进行了数学探索，使用一个简单的单层 transformer 研究了简单的潜在概念关联问题，理论和经验都表明 transformer 使用自注意力来收集信息并使用值矩阵进行联想记忆。