Answering complex natural language questions often necessitates multi-step
reasoning and integrating external information. Several systems have combined
knowledge retrieval with a large language model (LLM) to answer such questions.
These systems, however, suffer from various failure cases, and we cannot
directly train them end-to-end to fix such failures, as interaction with
external knowledge is non-differentiable. To address these deficiencies, we
define a ReAct-style LLM agent with the ability to reason and act upon external
knowledge. We further refine the agent through a ReST-like method that
iteratively trains on previous trajectories, employing growing-batch
reinforcement learning with AI feedback for continuous self-improvement and
self-distillation. Starting from a prompted large model and after just two
iterations of the algorithm, we can produce a fine-tuned small model that
achieves comparable performance on challenging compositional question-answering
benchmarks with two orders of magnitude fewer parameters.

通过 ReAct-style LLM 代理，我们可以迭代训练一个经过改进的小模型，用于具有挑战性的组合式问答基准测试，参数数量减少了两个数量级。