One way to enhance the reasoning capability of Large Language Models (LLMs)
is to conduct Supervised Fine-Tuning (SFT) using Chain-of-Thought (CoT)
annotations. This approach does not show sufficiently strong generalization
ability, however, because the training only relies on the given CoT data. In
math problem-solving, for example, there is usually only one annotated
reasoning path for each question in the training data. Intuitively, it would be
better for the algorithm to learn from multiple annotated reasoning paths given
a question. To address this issue, we propose a simple yet effective approach
called Reinforced Fine-Tuning (ReFT) to enhance the generalizability of
learning LLMs for reasoning, with math problem-solving as an example. ReFT
first warmups the model with SFT, and then employs on-line reinforcement
learning, specifically the PPO algorithm in this paper, to further fine-tune
the model, where an abundance of reasoning paths are automatically sampled
given the question and the rewards are naturally derived from the ground-truth
answers. Extensive experiments on GSM8K, MathQA, and SVAMP datasets show that
ReFT significantly outperforms SFT, and the performance can be potentially
further boosted by combining inference-time strategies such as majority voting
and re-ranking. Note that ReFT obtains the improvement by learning from the
same training questions as SFT, without relying on extra or augmented training
questions. This indicates a superior generalization ability for ReFT.

通过增强学习和在线增强学习的组合，提出了一种名为 ReFT 的简单而有效的方法来增强大型语言模型在推理中的泛化能力，以数学问题求解为例，通过学习多个标注的推理路径，显著提高了性能。