Language models have demonstrated remarkable performance in solving reasoning tasks; however, even the strongest models still occasionally make reasoning mistakes. Recently, there has been active research aimed at improving reasoning accuracy, particularly by using pretrained language models to "self-correct" their mistakes via multi-round prompting. In this paper, we follow this line of work but focus on understanding the usefulness of incorporating "error-correction" data directly into the pretraining stage. This data consists of erroneous solution steps immediately followed by their corrections. Using a synthetic math dataset, we show promising results: this type of pretrain data can help language models achieve higher reasoning accuracy directly (i.e., through simple auto-regression, without multi-round prompting) compared to pretraining on the same amount of error-free data. We also delve into many details, such as (1) how this approach differs from beam search, (2) how such data can be prepared, (3) whether masking is needed on the erroneous tokens, (4) the amount of error required, (5) whether such data can be deferred to the fine-tuning stage, and many others.

本研究解决了语言模型在推理任务中偶尔出现的错误问题。通过将“错误修正”数据直接引入预训练阶段，本文展示了该方法相比使用无误数据的预训练，能够显著提高语言模型的推理准确性，具有广泛的潜在影响。研究发现，此技术可实现单次自回归而无需多轮提示，从而简化推理过程。

语言模型的物理学：第二部分2.2，如何从小学数学问题的错误中学习