The self-improving ability of large language models (LLMs), enabled by
prompting them to analyze and revise their own outputs, has garnered
significant interest in recent research. However, this ability has been shown
to be absent and difficult to learn for smaller models, thus widening the
performance gap between state-of-the-art LLMs and more cost-effective and
faster ones. To reduce this gap, we introduce TriPosT, a training algorithm
that endows smaller models with such self-improvement ability, and show that
our approach can improve a LLaMA-7b's performance on math and reasoning tasks
by up to 7.13%. In contrast to prior work, we achieve this by using the smaller
model to interact with LLMs to collect feedback and improvements on its own
generations. We then replay this experience to train the small model. Our
experiments on four math and reasoning datasets show that the interactive
experience of learning from and correcting its own mistakes is crucial for
small models to improve their performance.

通过 TriPosT 训练算法，这篇论文介绍如何赋予更小的模型自我改进的能力，从而减小大型语言模型与成本效益更高、运行更快的模型之间的性能差距，并且通过与大型语言模型进行互动，收集反馈和改进，并将这一经验用于训练小模型，实验证明通过学习和纠正自己的错误对于小模型改进性能至关重要。