Increasing the size of a Transformer model does not always lead to enhanced
performance. This phenomenon cannot be explained by the empirical scaling laws.
Furthermore, improved generalization ability occurs as the model memorizes the
training samples. We present a theoretical framework that sheds light on the
memorization process and performance dynamics of transformer-based language
models. We model the behavior of Transformers with associative memories using
Hopfield networks, such that each transformer block effectively conducts an
approximate nearest-neighbor search. Based on this, we design an energy
function analogous to that in the modern continuous Hopfield network which
provides an insightful explanation for the attention mechanism. Using the
majorization-minimization technique, we construct a global energy function that
captures the layered architecture of the Transformer. Under specific
conditions, we show that the minimum achievable cross-entropy loss is bounded
from below by a constant approximately equal to 1. We substantiate our
theoretical results by conducting experiments with GPT-2 on various data sizes,
as well as training vanilla Transformers on a dataset of 2M tokens.

增加 Transformer 模型的大小并不总是导致性能提升，用经验缩放定律无法解释此现象。此外，模型记忆训练样本会改善泛化能力。我们提出了一个理论框架，揭示了基于 Transformer 的语言模型的记忆过程和性能动态。我们使用关联记忆的 Hopfield 网络来模拟 Transformer 的行为，使每个 Transformer 模块能够有效进行近似最近邻搜索。基于此，我们设计了一个能量函数，类似于现代连续 Hopfield 网络中的函数，对注意力机制提供了有见地的解释。利用最大化 - 最小化技术，我们构建了一个全局能量函数，捕捉了 Transformer 的层次结构。在特定条件下，我们证明了最小可达的交叉熵损失下界约为 1。通过对不同数据规模运行 GPT-2 实验证实了我们的理论结果，以及在一个包含 2M 令牌的数据集上训练 vanilla Transformers。