Current approaches for training Process Reward Models (PRMs) often involve breaking down responses into multiple reasoning steps using rule-based techniques, such as using predefined placeholder tokens or setting the reasoning step's length into a fixed size. These approaches overlook the fact that specific words do not typically mark true decision points in a text. To address this, we propose AdaptiveStep, a method that divides reasoning steps based on the model's confidence in predicting the next word. This division method provides more decision-making information at each step, enhancing downstream tasks, such as reward model learning. Moreover, our method does not require manual annotation. We demonstrate its effectiveness through experiments with AdaptiveStep-trained PRMs in mathematical reasoning and code generation tasks. Experimental results indicate that the outcome PRM achieves state-of-the-art Best-of-N performance, surpassing greedy search strategy with token-level value-guided decoding, while also reducing construction costs by over 30% compared to existing open-source PRMs. In addition, we provide a thorough analysis and case study on the PRM's performance, transferability, and generalization capabilities.

本研究解决了当前过程奖励模型（PRMs）训练中依据固定规则划分推理步骤的问题，提出了自适应步骤（AdaptiveStep）方法，通过模型在预测下一个词时的置信度来划分推理步骤。这一新方法在数学推理和代码生成任务中有效提升了奖励模型学习的效果，且在成本上较现有开源PRMs降低超过30%。

自适应步骤：通过模型置信度自动划分推理步骤