Training on model-generated synthetic data is a promising approach for
finetuning LLMs, but it remains unclear when it helps or hurts. In this paper,
we investigate this question for math reasoning via an empirical study,
followed by building a conceptual understanding of our observations. First, we
find that while the typical approach of finetuning a model on synthetic correct
or positive problem-solution pairs generated by capable models offers modest
performance gains, sampling more correct solutions from the finetuned learner
itself followed by subsequent fine-tuning on this self-generated data
$\textbf{doubles}$ the efficiency of the same synthetic problems. At the same
time, training on model-generated positives can amplify various spurious
correlations, resulting in flat or even inverse scaling trends as the amount of
data increases. Surprisingly, we find that several of these issues can be
addressed if we also utilize negative responses, i.e., model-generated
responses that are deemed incorrect by a final answer verifier. Crucially,
these negatives must be constructed such that the training can appropriately
recover the utility or advantage of each intermediate step in the negative
response. With this per-step scheme, we are able to attain consistent gains
over only positive data, attaining performance similar to amplifying the amount
of synthetic data by $\mathbf{8 \times}$. We show that training on per-step
negatives can help to unlearn spurious correlations in the positive data, and
is equivalent to advantage-weighted reinforcement learning (RL), implying that
it inherits robustness benefits of RL over imitating positive data alone.

通过经验研究，我们发现利用模型生成的合成数据进行训练可以提高数学推理的性能，但是通过添加负回答可以进一步增强效果，并去除其中的虚假相关性。

通过错误的合成数据应用 RL 技术提高数理推理任务效率八倍

RL on Incorrect Synthetic Data Scales the Efficiency of LLM Math  Reasoning by Eight-Fold

During recent years, active learning has evolved into a popular paradigm for
utilizing user's feedback to improve accuracy of learning algorithms. Active
learning works by selecting the most informative sample among unlabeled data
and querying the label of that point from user. Many different methods such as
uncertainty sampling and minimum risk sampling have been utilized to select the
most informative sample in active learning. Although many active learning
algorithms have been proposed so far, most of them work with binary or
multi-class classification problems and therefore can not be applied to
problems in which only samples from one class as well as a set of unlabeled
data are available.
Such problems arise in many real-world situations and are known as the
problem of learning from positive and unlabeled data. In this paper we propose
an active learning algorithm that can work when only samples of one class as
well as a set of unlabelled data are available. Our method works by separately
estimating probability desnity of positive and unlabeled points and then
computing expected value of informativeness to get rid of a hyper-parameter and
have a better measure of informativeness./ Experiments and empirical analysis
show promising results compared to other similar methods.

提出了一种能够在只有单类样本和无标签数据的情况下运作的主动学习算法，该方法通过分别估计正样本和未标记点的概率密度，计算信息性的期望值以获取更好的信息性度量。与其他类似方法相比，实验和经验分析结果表现出有前途的结果。