Self-training can potentially improve the performance of language agents
without relying on demonstrations from humans or stronger models. The general
process involves generating samples from a model, evaluating their quality, and
updating the model by training on high-quality samples. However, self-training
can face limitations because achieving good performance requires a good amount
of high-quality samples, yet relying solely on model sampling for obtaining
such samples can be inefficient. In addition, these methods often disregard
low-quality samples, failing to leverage them effectively. To address these
limitations, we present Reflection-Reinforced Self-Training (Re-ReST), which
leverages a reflection model to refine low-quality samples and subsequently
uses these improved samples to augment self-training. The reflection model
takes both the model output and feedback from an external environment (e.g.,
unit test results in code generation) as inputs and produces improved samples
as outputs. By employing this technique, we effectively enhance the quality of
inferior samples, and enrich the self-training dataset with higher-quality
samples efficiently. We perform extensive experiments on open-source language
agents across tasks, including multi-hop question answering, sequential
decision-making, code generation, visual question answering, and text-to-image
generation. Results demonstrate improvements over self-training baselines
across settings. Moreover, ablation studies confirm the reflection model's
efficiency in generating quality self-training samples and its compatibility
with self-consistency decoding.

Reflection-Reinforced Self-Training (Re-ReST) leverages a reflection model to refine low-quality samples and augment self-training, enhancing the quality of samples efficiently.