In this paper, we propose R$^3$: Learning Reasoning through Reverse Curriculum Reinforcement Learning (RL), a novel method that employs only outcome supervision to achieve the benefits of process supervision for large language models. The core challenge in applying RL to complex reasoning is to identify a sequence of actions that result in positive rewards and provide appropriate supervision for optimization. Outcome supervision provides sparse rewards for final results without identifying error locations, whereas process supervision offers step-wise rewards but requires extensive manual annotation. R$^3$ overcomes these limitations by learning from correct demonstrations. Specifically, R$^3$ progressively slides the start state of reasoning from a demonstration's end to its beginning, facilitating easier model exploration at all stages. Thus, R$^3$ establishes a step-wise curriculum, allowing outcome supervision to offer step-level signals and precisely pinpoint errors. Using Llama2-7B, our method surpasses RL baseline on eight reasoning tasks by $4.1$ points on average. Notebaly, in program-based reasoning on GSM8K, it exceeds the baseline by $4.2$ points across three backbone models, and without any extra data, Codellama-7B + R$^3$ performs comparable to larger models or closed-source models.

本研究提出了一种名为R^3的学习推理的逆向课程强化学习方法，该方法仅借助结果监督实现大型语言模型的过程监督的益处。该方法通过学习正确演示，使用逐步课程逐渐滑动推理起始状态，以便在所有阶段更容易地进行模型探索，从而允许结果监督提供逐步信号并准确定位错误。使用Llama2-7B，在八个推理任务上，我们的方法平均超过基于强化学习的基线4.1个点。值得注意的是，在基于程序的推理任务GSM8K上，与不使用任何额外数据的基线相比，Codellama-7B + R^3在三个骨干模型上的表现相当于更大的模型或闭源模型。

通过逆序课程强化学习训练大规模语言模型