Transformers exhibit impressive capabilities but are often regarded as black
boxes due to challenges in understanding the complex nonlinear relationships
between features. Interpreting machine learning models is of paramount
importance to mitigate risks, and mechanistic interpretability is in particular
of current interest as it opens up a window for guiding manual modifications
and reverse-engineering solutions. In this work, we introduce contextual
decomposition for transformers (CD-T), extending a prior work on CD for RNNs
and CNNs, to address mechanistic interpretation computationally efficiently.
CD-T is a flexible interpretation method for transformers. It can capture
contributions of combinations of input features or source internal components
(e.g. attention heads, feed-forward networks) to (1) final predictions or (2)
the output of any target internal component. Using CD-T, we propose a novel
algorithm for circuit discovery. On a real-world pathology report
classification task: we show CD-T distills a more faithful circuit of attention
heads with improved computational efficiency (speed up 2x) than a prior
benchmark, path patching. As a versatile interpretation method, CD-T also
exhibits exceptional capabilities for local interpretations. CD-T is shown to
reliably find words and phrases of contrasting sentiment/topic on SST-2 and
AGNews datasets. Through human experiments, we demonstrate CD-T enables users
to identify the more accurate of two models and to better trust a model's
outputs compared to alternative interpretation methods such as SHAP and LIME.

本文介绍了一种用于 transformers 的上下文分解的解释方法，可以捕捉不同输入特征或源内部组件对最终预测或目标内部组件输出的贡献，用于电路发现等任务，并通过实验验证了其在局部解释方面的出色能力。

基于上下文分解的 Transformer 中的机制解释

Mechanistic Interpretation through Contextual Decomposition in  Transformers

In the field of language model interpretability, circuit discovery is gaining
popularity. Despite this, the true meaning of these circuits remain largely
unanswered. We introduce a novel method to learn their meanings as a holistic
object through the lens of knowledge editing. We extract circuits in the
GPT2-XL model using diverse text classification datasets, and use hierarchical
relations datasets to explore knowledge editing in the circuits. Our findings
indicate that these circuits contain entity knowledge but resist new knowledge
more than complementary circuits during knowledge editing. Additionally, we
examine the impact of circuit size, discovering that an ideal "theoretical
circuit" where essential knowledge is concentrated likely incorporates more
than 5% but less than 50% of the model's parameters. We also assess the overlap
between circuits from different datasets, finding moderate similarities. What
constitutes these circuits, then? We find that up to 60% of the circuits
consist of layer normalization modules rather than attention or MLP modules,
adding evidence to the ongoing debates regarding knowledge localization. In
summary, our findings offer new insights into the functions of the circuits,
and introduce research directions for further interpretability and safety
research of language models.

通过知识编辑的视角，我们介绍了一种学习语言模型电路的新方法。通过在 GPT2-XL 模型中提取电路，使用多样的文本分类数据集和层次关系数据集，我们发现这些电路包含实体知识，但在知识编辑中对新知识的抵抗力高于互补电路。此外，我们研究了电路尺寸的影响，发现理想的电路可能包含模型参数的 5% 至 50%。我们还检查了不同数据集中电路之间的重叠情况，发现中等相似性。总之，我们的研究结果为电路的功能提供了新的见解，并引入了进一步解释性和安全性研究的方向。