Data plays a fundamental role in the training of Large Language Models
(LLMs). While attention has been paid to the collection and composition of
datasets, determining the data sampling strategy in training remains an open
question. Most LLMs are trained with a simple strategy, random sampling.
However, this sampling strategy ignores the unbalanced nature of training data
distribution, which can be sub-optimal. In this paper, we propose ClusterClip
Sampling to balance the text distribution of training data for better model
training. Specifically, ClusterClip Sampling utilizes data clustering to
reflect the data distribution of the training set and balances the common
samples and rare samples during training based on the cluster results. A
repetition clip operation is introduced to mitigate the overfitting issue led
by samples from certain clusters. Extensive experiments validate the
effectiveness of ClusterClip Sampling, which outperforms random sampling and
other cluster-based sampling variants under various training datasets and large
language models.

在这篇论文中，我们提出了一种名为 ClusterClip Sampling 的数据采样策略，利用数据聚类来平衡训练数据的文本分布以获得更好的模型训练效果。通过基于聚类结果对训练过程中的常见样本和罕见样本进行平衡，以及引入重复剪裁操作来减轻由于特定聚类样本引起的过拟合问题。大量的实验证明了 ClusterClip Sampling 的有效性，在各种训练数据集和大型语言模型上的表现优于随机采样和其他基于聚类的采样变体。