Neural network models have achieved high performance on a wide variety of
complex tasks, but the algorithms that they implement are notoriously difficult
to interpret. In order to understand these algorithms, it is often necessary to
hypothesize intermediate variables involved in the network's computation. For
example, does a language model depend on particular syntactic properties when
generating a sentence? However, existing analysis tools make it difficult to
test hypotheses of this type. We propose a new analysis technique -- circuit
probing -- that automatically uncovers low-level circuits that compute
hypothesized intermediate variables. This enables causal analysis through
targeted ablation at the level of model parameters. We apply this method to
models trained on simple arithmetic tasks, demonstrating its effectiveness at
(1) deciphering the algorithms that models have learned, (2) revealing modular
structure within a model, and (3) tracking the development of circuits over
training. We compare circuit probing to other methods across these three
experiments, and find it on par or more effective than existing analysis
methods. Finally, we demonstrate circuit probing on a real-world use case,
uncovering circuits that are responsible for subject-verb agreement and
reflexive anaphora in GPT2-Small and Medium.

神经网络模型在各种复杂任务上取得了高性能，但它们所实现的算法往往难以解释。我们提出了一种新的分析技术 —— 电路探测，通过自动发现计算假定的中间变量的低层电路，实现了对模型参数级别的有针对性的切割，从而开展因果分析。我们对简单算术任务上的模型应用了这种方法，证明了它在（1）解密模型所学习的算法，（2）揭示模型内部结构以及（3）追踪电路在训练过程中的发展方面的有效性。我们将电路探测与其他方法在这三个实验中进行比较，发现它在效果上与现有的分析方法相当甚至更加有效。最后，我们在一个真实的应用案例中演示了电路探测的应用，发现了在 GPT2-Small 和 Medium 模型中负责主谓一致和反身指代的电路。