Mechanistic interpretability aims to understand the behavior of neural
networks by reverse-engineering their internal computations. However, current
methods struggle to find clear interpretations of neural network activations
because a decomposition of activations into computational features is missing.
Individual neurons or model components do not cleanly correspond to distinct
features or functions. We present a novel interpretability method that aims to
overcome this limitation by transforming the activations of the network into a
new basis - the Local Interaction Basis (LIB). LIB aims to identify
computational features by removing irrelevant activations and interactions. Our
method drops irrelevant activation directions and aligns the basis with the
singular vectors of the Jacobian matrix between adjacent layers. It also scales
features based on their importance for downstream computation, producing an
interaction graph that shows all computationally-relevant features and
interactions in a model. We evaluate the effectiveness of LIB on modular
addition and CIFAR-10 models, finding that it identifies more
computationally-relevant features that interact more sparsely, compared to
principal component analysis. However, LIB does not yield substantial
improvements in interpretability or interaction sparsity when applied to
language models. We conclude that LIB is a promising theory-driven approach for
analyzing neural networks, but in its current form is not applicable to large
language models.

通过将神经网络的激活转换为新的基础 - 局部互动基础（LIB），我们提出了一种新颖的可解释性方法，旨在识别计算特征，通过消除无关的激活和相互作用，以及基于它们对下游计算的重要性对特征进行缩放，生成显示模型中所有计算相关特征和相互作用的交互图。