Although it is known that transformer language models (LMs) pass features
from early layers to later layers, it is not well understood how this
information is represented and routed by the model. By analyzing particular
mechanism LMs use to accomplish this, we find that it is also used to recall
items from a list, and show that this mechanism can explain an otherwise
arbitrary-seeming sensitivity of the model to the order of items in the prompt.
Specifically, we find that models write into low-rank subspaces of the residual
stream to represent features which are then read out by specific later layers,
forming low-rank communication channels between layers. By decomposing
attention head weight matrices with the Singular Value Decomposition (SVD), we
find that previously described interactions between heads separated by one or
more layers can be predicted via analysis of their weight matrices. We show
that it is possible to manipulate the internal model representations as well as
edit model weights based on the mechanism we discover in order to significantly
improve performance on our synthetic Laundry List task, which requires recall
from a list, often improving task accuracy by over 20%. Our analysis reveals a
surprisingly intricate interpretable structure learned from language model
pretraining, and helps us understand why sophisticated LMs sometimes fail in
simple domains, facilitating future analysis of more complex behaviors.

通过分析 Transformer 语言模型中的机制，研究揭示了其通过低秩通信信道在不同层之间传递和存储特征的方式，通过分解注意力头权重矩阵进而预测层间相互作用的方法，以及利用该机制改善内部模型表示和权重来提升任务性能的结构学习，为进一步分析复杂行为提供帮助。