Interpretability methods aim to understand the algorithm implemented by a trained model (e.g., a Transofmer) by examining various aspects of the model, such as the weight matrices or the attention patterns. In this work, through a combination of theoretical results and carefully controlled experiments on synthetic data, we take a critical view of methods that exclusively focus on individual parts of the model, rather than consider the network as a whole. We consider a simple synthetic setup of learning a (bounded) Dyck language. Theoretically, we show that the set of models that (exactly or approximately) solve this task satisfy a structural characterization derived from ideas in formal languages (the pumping lemma). We use this characterization to show that the set of optima is qualitatively rich; in particular, the attention pattern of a single layer can be ``nearly randomized'', while preserving the functionality of the network. We also show via extensive experiments that these constructions are not merely a theoretical artifact: even after severely constraining the architecture of the model, vastly different solutions can be reached via standard training. Thus, interpretability claims based on inspecting individual heads or weight matrices in the Transformer can be misleading.

研究关注于解释性方法，通过检查模型的不同方面（例如权重矩阵或注意力模式）来理解训练模型（例如Transformer）实现的算法。通过理论结果和对合成数据的仔细控制实验的组合，我们对专注于模型的个别部分而不是整个网络的方法提出了批判性观点。我们使用学习（有界）Dyck语言的简单合成设置来理论上展示了解决此任务的模型集合满足的结构特征（基于形式语言的思想，即泵引理）并证明了最佳解集合具有丰富的定性特点；特定情况下，单层的注意力模式可以“几乎随机化”，同时保持网络的功能性。通过大量实验证明了这些结论并不仅仅是理论产物：即使在严格约束模型架构的情况下，经过标准训练也可以得到截然不同的解决方案。因此，基于检查Transformer中的个别头部或权重矩阵的解释性声明可能是误导性的。

基于有界Dyck文法的研究案例：变压器使用短视方法不可解释