Recent work has shown that language models (LMs) have strong multi-step
(i.e., procedural) reasoning capabilities. However, it is unclear whether LMs
perform these tasks by cheating with answers memorized from pretraining corpus,
or, via a multi-step reasoning mechanism. In this paper, we try to answer this
question by exploring a mechanistic interpretation of LMs for multi-step
reasoning tasks. Concretely, we hypothesize that the LM implicitly embeds a
reasoning tree resembling the correct reasoning process within it. We test this
hypothesis by introducing a new probing approach (called MechanisticProbe) that
recovers the reasoning tree from the model's attention patterns. We use our
probe to analyze two LMs: GPT-2 on a synthetic task (k-th smallest element),
and LLaMA on two simple language-based reasoning tasks (ProofWriter & AI2
Reasoning Challenge). We show that MechanisticProbe is able to detect the
information of the reasoning tree from the model's attentions for most
examples, suggesting that the LM indeed is going through a process of
multi-step reasoning within its architecture in many cases.

在这篇论文中，我们通过探索一种机械化方法来回答语言模型在多步推理任务中是通过抄袭预训练语料库中记忆的答案还是通过多步推理机制来完成这些任务的疑问。我们引入了一种新的探测方法（称为机械化探测器），从模型的注意力模式中恢复推理树，用于分析两个语言模型：GPT-2 在一个合成任务（第 k 个最小元素）上以及 LLaMA 在两个简单的基于语言的推理任务（ProofWriter 和 AI2 推理挑战）上。我们展示了机械化探测器能够在大多数示例中从模型的注意力中检测到推理树的信息，这表明在许多情况下语言模型确实在其架构中经历了一个多步推理的过程。