Prevalent reinforcement learning~(RL) methods for fine-tuning LLM reasoners, such as GRPO or Leave-one-out PPO, abandon the learned value function in favor of empirically estimated returns. This hinders test-time compute scaling that relies on using the value-function for verification. In this work, we propose RL$^V$ that augments any ``value-free'' RL method by jointly training the LLM as both a reasoner and a generative verifier using RL-generated data, adding verification capabilities without significant overhead. Empirically, RL$^V$ boosts MATH accuracy by over 20\% with parallel sampling and enables $8-32\times$ efficient test-time compute scaling compared to the base RL method. RL$^V$ also exhibits strong generalization capabilities for both easy-to-hard and out-of-domain tasks. Furthermore, RL$^V$ achieves $1.2-1.6\times$ higher performance when jointly scaling parallel and sequential test-time compute with a long reasoning R1 model.

本研究解决了现有强化学习方法中，未能有效利用学习到的价值函数的问题。通过提出RL$^V$，将大语言模型同时训练为推理器和生成验证器，从而增强验证能力，显著提高了MATH任务的准确率，并实现了高达$8-32\times$的测试时间计算效率提升，展示了出色的泛化能力和性能。 

将价值重新引入强化学习：通过统一大语言模型推理器与验证器实现更好的测试时间缩放