Process supervision, using a trained verifier to evaluate the intermediate
steps generated by reasoner, has demonstrated significant improvements in
multi-step problem solving. In this paper, to avoid expensive human annotation
effort on the verifier training data, we introduce Model-induced Process
Supervision (MiPS), a novel method for automating data curation. MiPS annotates
an intermediate step by sampling completions of this solution through the
reasoning model, and obtaining an accuracy defined as the proportion of correct
completions. Errors in the reasoner would cause MiPS to underestimate the
accuracy of intermediate steps, therefore, we suggest and empirically show that
verification focusing on high predicted scores of the verifier shall be
preferred over that of low predicted scores, contrary to prior work. Our
approach significantly improves the performance of PaLM 2 on math and coding
tasks (accuracy +0.67% on GSM8K, +4.16% on MATH, +0.92% on MBPP compared with
an output supervision trained verifier). Additionally, our study demonstrates
that the verifier exhibits strong generalization ability across different
reasoning models.

通过使用训练过的验证器评估推理器生成的中间步骤，过程监督在多步问题解决中展示出了显著的改进。为了避免在验证器训练数据上进行昂贵的人工注释，本文引入了一种新颖的自动化数据整理方法，即模型引导的过程监督（MiPS）。MiPS 通过对推理模型的解决方案的样本完成进行注释，并获得一个准确度，该准确度被定义为正确完成的比例，来注释中间步骤。推理器中的错误会导致 MiPS 低估中间步骤的准确度，因此我们建议并经验证明，与以往研究相反，应该优先选择验证器的高预测分数而不是低预测分数。我们的方法显著提高了 PaLM 2 在数学和编码任务上的性能（与输出监督训练的验证器相比，GSM8K 的准确度提高了 0.67％，MATH 的准确度提高了 4.16％，MBPP 的准确度提高了 0.92％）。此外，我们的研究表明，验证器在不同推理模型之间具有很强的泛化能力。