We provide a comprehensive analysis of simple transformer models trained on the histogram task, where the goal is to count the occurrences of each item in the input sequence from a fixed alphabet. Despite its apparent simplicity, this task exhibits a rich phenomenology that allows us to characterize how different architectural components contribute towards the emergence of distinct algorithmic solutions. In particular, we showcase the existence of two qualitatively different mechanisms that implement a solution, relation- and inventory-based counting. Which solution a model can implement depends non-trivially on the precise choice of the attention mechanism, activation function, memorization capacity and the presence of a beginning-of-sequence token. By introspecting learned models on the counting task, we find evidence for the formation of both mechanisms. From a broader perspective, our analysis offers a framework to understand how the interaction of different architectural components of transformer models shapes diverse algorithmic solutions and approximations.

我们全面分析了在直方图任务中训练的简单Transformer模型，该任务的目标是计算来自固定字母表的输入序列中每个项的出现次数。尽管表面上看起来很简单，但该任务表现出丰富的现象学，使我们能够描述不同的架构组件如何对不同的算法解决方案的出现做出贡献。我们介绍了实现解决方案的两种定性不同机制：关系和库存法计数。模型能够实现哪种解决方案取决于注意机制、激活函数、记忆容量的精确选择以及是否存在起始序列令牌等因素，通过对计数任务的学习模型进行反思，我们发现了形成两种机制的证据。从更广泛的角度来看，我们的分析提供了一个框架，用于理解Transformer模型的不同架构组件之间的相互作用如何塑造多样化的算法解决方案和近似。

小型Transformer中的计数理解：注意力与前馈层之间的相互作用