Direct Preference Optimization (DPO) has proven effective at improving the
performance of large language models (LLMs) on downstream tasks such as
reasoning and alignment. In this work, we propose Step-Controlled DPO (SCDPO),
a method for automatically providing stepwise error supervision by creating
negative samples of mathematical reasoning rationales that start making errors
at a specified step. By applying these samples in DPO training, SCDPO can
better align the model to understand reasoning errors and output accurate
reasoning steps. We apply SCDPO to both code-integrated and chain-of-thought
solutions, empirically showing that it consistently improves the performance
compared to naive DPO on three different SFT models, including one existing SFT
model and two models we finetuned. Qualitative analysis of the credit
assignment of SCDPO and DPO demonstrates the effectiveness of SCDPO at
identifying errors in mathematical solutions. We then apply SCDPO to an
InternLM2-20B model, resulting in a 20B model that achieves high scores of
88.5% on GSM8K and 58.1% on MATH, rivaling all other open-source LLMs, showing
the great potential of our method.

提出了一种名为 Step-Controlled DPO (SCDPO) 的方法，在大型语言模型 (LLM) 上应用全自动逐步错误监督，从而改善其在推理和对齐等下游任务上的性能。以此方法应用于数学解决方案，通过在 DPO 训练中采用负样本，SCDPO 能够更好地调整模型以理解推理错误并输出准确的推理步骤。该方法在不同的 SFT 模型上得到了改进，并展示了其在识别数学解决方案错误方面的有效性。最终应用 SCDPO 于一个 InternLM2-20B 模型，取得了 88.5% 的 GSM8K 得分和 58.1% 的 MATH 得分，与所有其他开源 LLM 相媲美，展示了这一方法的巨大潜力。