There are a number of diverging hypotheses about the neural text degeneration
problem, i.e., generating repetitive and dull loops, which makes this problem
both interesting and confusing. In this work, we aim to advance our
understanding by presenting a straightforward and fundamental explanation from
the data perspective. Our preliminary investigation reveals a strong
correlation between the degeneration issue and the presence of repetitions in
training data. Subsequent experiments also demonstrate that by selectively
dropping out the attention to repetitive words in training data, degeneration
can be significantly minimized. Furthermore, our empirical analysis illustrates
that prior works addressing the degeneration issue from various standpoints,
such as the high-inflow words, the likelihood objective, and the
self-reinforcement phenomenon, can be interpreted by one simple explanation.
That is, penalizing the repetitions in training data is a common and
fundamental factor for their effectiveness. Moreover, our experiments reveal
that penalizing the repetitions in training data remains critical even when
considering larger model sizes and instruction tuning.

我们通过数据角度的初步研究发现，训练数据中重复出现的词语与文本退化问题之间存在强烈的相关性，通过有选择地消除训练数据中重复词语的注意力，可以显著减少退化现象，并且惩罚训练数据中的重复部分是解决退化问题的一个普遍而基本的因素。