This study investigates the consequences of training large language models
(LLMs) on synthetic data generated by their predecessors, an increasingly
prevalent practice aimed at addressing the limited supply of human-generated
training data. Diverging from the usual emphasis on performance metrics, we
focus on the impact of this training methodology on linguistic diversity,
especially when conducted recursively over time. To assess this, we developed a
set of novel metrics targeting lexical, syntactic, and semantic diversity,
applying them in recursive fine-tuning experiments across various natural
language generation tasks. Our findings reveal a marked decrease in the
diversity of the models' outputs through successive iterations. This trend
underscores the potential risks of training LLMs on predecessor-generated text,
particularly concerning the preservation of linguistic richness. Our study
highlights the need for careful consideration of the long-term effects of such
training approaches on the linguistic capabilities of LLMs.

研究通过以前辈生成的合成数据对大型语言模型进行训练的后果，重点关注这种训练方法对语言多样性的影响，特别是在逐步迭代的过程中。通过开展递归微调实验，应用一系列针对词汇、句法和语义多样性的新型度量标准，我们的研究发现模型输出的多样性在连续迭代中显著降低。这一趋势强调了在训练大型语言模型时使用前辈生成文本的潜在风险，特别是涉及保留语言丰富性方面。我们的研究突出了需要仔细考虑这种训练方法对大型语言模型的语言能力所产生的长期影响。