Large language models (LLMs) have been shown to perform better when asked to
reason step-by-step before answering a question. However, it is unclear to what
degree the model's final answer is faithful to the stated reasoning steps. In
this paper, we perform a causal mediation analysis on twelve LLMs to examine
how intermediate reasoning steps generated by the LLM influence the final
outcome and find that LLMs do not reliably use their intermediate reasoning
steps when generating an answer. To address this issue, we introduce FRODO, a
framework to tailor small-sized LMs to generate correct reasoning steps and
robustly reason over these steps. FRODO consists of an inference module that
learns to generate correct reasoning steps using an implicit causal reward
function and a reasoning module that learns to faithfully reason over these
intermediate inferences using a counterfactual and causal preference objective.
Our experiments show that FRODO significantly outperforms four competitive
baselines. Furthermore, FRODO improves the robustness and generalization
ability of the reasoning LM, yielding higher performance on out-of-distribution
test sets. Finally, we find that FRODO's rationales are more faithful to its
final answer predictions than standard supervised fine-tuning.

通过对十二个大型语言模型进行因果中介分析，本文发现大型语言模型在生成最终答案时并不可靠地使用中间推理步骤。为了解决这个问题，我们引入了 FRODO 框架，该框架通过使用隐式因果奖励函数生成正确的推理步骤，以及通过因果和对抗优化目标忠实地进行推理。实验证明，FRODO 显著优于其他四个基线方法，提高了推理语言模型的鲁棒性和泛化能力，在分布外测试集上表现更好。最后，我们发现 FRODO 的解释对最终答案的预测更加可靠。