Pretrained language models have shown superior performance on many natural
language processing tasks, yet they still struggle at multi-step formal
reasoning tasks like grade school math problems. One key challenge of
finetuning them to solve such math reasoning problems is that many existing
datasets only contain one reference solution for each problem, despite the fact
that there are often alternative solutions resembling different reasoning paths
to the final answer. This way, the finetuned models are biased towards the
limited reference solutions, which limits their generalization to unseen
examples. To mitigate this issue, we propose to let the model perform sampling
during training and learn from both self-sampled fully-correct solutions, which
yield the correct answer upon execution, and partially-correct solutions, whose
intermediate state matches an intermediate state of a known correct solution.
We show that our use of self-sampled correct and partially-correct solutions
can benefit learning and help guide the sampling process, leading to more
efficient exploration of the solution space. Additionally, we explore various
training objectives to support learning from multiple solutions per example and
find they greatly affect the performance. Experiments on two math reasoning
datasets show the effectiveness of our method compared to learning from a
single reference solution with MLE, where we improve PASS@100 from 35.5% to
44.5% for GSM8K, and 27.6% to 36.2% PASS@80 for MathQA. Such improvements are
also consistent across different model sizes. Our code is available at
this https URL

本文提出使用预先训练的语言模型和部分正确的解决方案来解决数学推理问题的训练方法，并探索了各种训练目标对性能的影响。部分正确的解决方案有助于模型对解决方案空间进行更有效的探索。在两个数学推理数据集上的实验显示出方法的有效性。