The phenomenon of model collapse, introduced in (Shumailov et al., 2023),
refers to the deterioration in performance that occurs when new models are
trained on synthetic data generated from previously trained models. This
recursive training loop makes the tails of the original distribution disappear,
thereby making future-generation models forget about the initial (real)
distribution. With the aim of rigorously understanding model collapse in
language models, we consider in this paper a statistical model that allows us
to characterize the impact of various recursive training scenarios.
Specifically, we demonstrate that model collapse cannot be avoided when
training solely on synthetic data. However, when mixing both real and synthetic
data, we provide an estimate of a maximal amount of synthetic data below which
model collapse can eventually be avoided. Our theoretical conclusions are
further supported by empirical validations.

模型崩溃是指在使用之前训练的模型生成的合成数据训练新模型时性能下降的现象，本文通过统计模型对各种递归训练情景的影响进行了深入研究，发现当仅使用合成数据进行训练时无法避免模型崩溃，但当混合使用真实数据和合成数据时，我们提供了在合成数据量不超过一定阈值时模型崩溃可以最终避免的估计值，我们的理论结论得到了经验验证的支持。

合成数据训练的糟糕程度：语言模型崩溃的统计分析

How Bad is Training on Synthetic Data? A Statistical Analysis of  Language Model Collapse

Machine learning models may inadvertently memorize sensitive, unauthorized,
or malicious data, posing risks of privacy violations, security breaches, and
performance deterioration. To address these issues, machine unlearning has
emerged as a critical technique to selectively remove specific training data
points' influence on trained models. This paper provides a comprehensive
taxonomy and analysis of machine unlearning research. We categorize existing
research into exact unlearning that algorithmically removes data influence
entirely and approximate unlearning that efficiently minimizes influence
through limited parameter updates. By reviewing the state-of-the-art solutions,
we critically discuss their advantages and limitations. Furthermore, we propose
future directions to advance machine unlearning and establish it as an
essential capability for trustworthy and adaptive machine learning. This paper
provides researchers with a roadmap of open problems, encouraging impactful
contributions to address real-world needs for selective data removal.

机器遗忘是一项关键技术，用于选择性地移除训练数据点对训练模型的影响。本文提供了机器遗忘研究的全面分类和分析，并回顾了最新的解决方案，讨论了其优势和限制，并提出了未来的发展方向，以推动机器遗忘作为一项必要的可信和自适应机器学习能力。

机器取消学习：解决方案与挑战

Machine Unlearning: Solutions and Challenges

Self-modification of agents embedded in complex environments is hard to
avoid, whether it happens via direct means (e.g. own code modification) or
indirectly (e.g. influencing the operator, exploiting bugs or the environment).
It has been argued that intelligent agents have an incentive to avoid modifying
their utility function so that their future instances work towards the same
goals.
Everitt et al. (2016) formally show that providing an option to self-modify
is harmless for perfectly rational agents. We show that this result is no
longer true for agents with bounded rationality. In such agents,
self-modification may cause exponential deterioration in performance and
gradual misalignment of a previously aligned agent. We investigate how the size
of this effect depends on the type and magnitude of imperfections in the
agent's rationality (1-4 below). We also discuss model assumptions and the
wider problem and framing space.
We examine four ways in which an agent can be bounded-rational: it either (1)
doesn't always choose the optimal action, (2) is not perfectly aligned with
human values, (3) has an inaccurate model of the environment, or (4) uses the
wrong temporal discounting factor. We show that while in the cases (2)-(4) the
misalignment caused by the agent's imperfection does not increase over time,
with (1) the misalignment may grow exponentially.

我们调查了受限理性的代理如何随着时间的推移受到自我修改的负面影响，分析了其大小与代理的理性失误的类型和严重程度之间的关系，其中特别指出在选择非最佳行为的情况下可能会出现指数级错误对齐的情况。