Process Reward Models (PRMs) provide step-level supervision to large language models (LLMs), but scaling up training data annotation remains challenging for both humans and LLMs. To address this limitation, we propose an active learning approach, ActPRM, which proactively selects the most uncertain samples for training, substantially reducing labeling costs. During training, we use the PRM to estimate uncertainty after the forward pass, retaining only highly uncertain data. A capable yet costly reasoning model then labels this data. Then we compute the loss with respect to the labels and update the PRM's weights. We compare ActPRM vs. vanilla fine-tuning, on a pool-based active learning setting, demonstrating that ActPRM reduces 50% annotation, but achieving the comparable or even better performance. Beyond annotation efficiency, we further advance the actively trained PRM by filtering over 1M+ math reasoning trajectories with ActPRM, retaining 60% of the data. A subsequent training on this selected dataset yields a new state-of-the-art (SOTA) PRM on ProcessBench (75.0%) and PRMBench (65.5%) compared with same sized models.

本研究针对过程奖励模型在大规模语言模型训练中的标注数据获取困难进行了探讨，提出了主动学习方法ActPRM，能够主动选择最不确定的样本进行训练，从而显著降低标注成本。研究结果显示，ActPRM在减少50%标注的同时，取得了相当甚至更好的性能，并在ProcessBench和PRMBench上创造了新的最先进的效果。

通过主动学习实现高效的过程奖励模型训练