Data plays a fundamental role in the training of Large Language Models
(LLMs). While attention has been paid to the collection and composition of
datasets, determining the data sampling strategy in training remains an open
question. Most LLMs are trained with a simple strategy, random sampling.
However, this sampling strategy ignores the unbalanced nature of training data
distribution, which can be sub-optimal. In this paper, we propose ClusterClip
Sampling to balance the text distribution of training data for better model
training. Specifically, ClusterClip Sampling utilizes data clustering to
reflect the data distribution of the training set and balances the common
samples and rare samples during training based on the cluster results. A
repetition clip operation is introduced to mitigate the overfitting issue led
by samples from certain clusters. Extensive experiments validate the
effectiveness of ClusterClip Sampling, which outperforms random sampling and
other cluster-based sampling variants under various training datasets and large
language models.

在这篇论文中，我们提出了一种名为 ClusterClip Sampling 的数据采样策略，利用数据聚类来平衡训练数据的文本分布以获得更好的模型训练效果。通过基于聚类结果对训练过程中的常见样本和罕见样本进行平衡，以及引入重复剪裁操作来减轻由于特定聚类样本引起的过拟合问题。大量的实验证明了 ClusterClip Sampling 的有效性，在各种训练数据集和大型语言模型上的表现优于随机采样和其他基于聚类的采样变体。

使用聚类进行语言模型训练的平衡数据采样

Balanced Data Sampling for Language Model Training with Clustering

Neural network language model (NNLM) plays an essential role in automatic
speech recognition (ASR) systems, especially in adaptation tasks when text-only
data is available. In practice, an NNLM is typically trained on a combination
of data sampled from multiple corpora. Thus, the data sampling strategy is
important to the adaptation performance. Most existing works focus on designing
static sampling strategies. However, each corpus may show varying impacts at
different NNLM training stages. In this paper, we introduce a novel adaptive
multi-corpora training algorithm that dynamically learns and adjusts the
sampling probability of each corpus along the training process. The algorithm
is robust to corpora sizes and domain relevance. Compared with static sampling
strategy baselines, the proposed approach yields remarkable improvement by
achieving up to relative 7% and 9% word error rate (WER) reductions on
in-domain and out-of-domain adaptation tasks, respectively.

本文提出了一种新颖的自适应多语料库训练算法，该算法可以动态学习和调整每个语料库的采样概率，相较于静态采样策略可使得领域内和领域外的适应任务分别取得相对 7% 和 9% 的字词错误率降低。

语音识别自适应多语料语言模型训练

Adaptive Multi-Corpora Language Model Training for Speech Recognition

Interactive image restoration aims to generate restored images by adjusting a
controlling coefficient which determines the restoration level. Previous works
are restricted in modulating image with a single coefficient. However, real
images always contain multiple types of degradation, which cannot be well
determined by one coefficient. To make a step forward, this paper presents a
new problem setup, called multi-dimension (MD) modulation, which aims at
modulating output effects across multiple degradation types and levels.
Compared with the previous single-dimension (SD) modulation, the MD is setup to
handle multiple degradations adaptively and relief unbalanced learning problem
in different degradations. We also propose a deep architecture - CResMD with
newly introduced controllable residual connections for multi-dimension
modulation. Specifically, we add a controlling variable on the conventional
residual connection to allow a weighted summation of input and residual. The
values of these weights are generated by another condition network. We further
propose a new data sampling strategy based on beta distribution to balance
different degradation types and levels. With corrupted image and degradation
information as inputs, the network can output the corresponding restored image.
By tweaking the condition vector, users can control the output effects in MD
space at test time. Extensive experiments demonstrate that the proposed CResMD
achieve excellent performance on both SD and MD modulation tasks. Code is
available at this https URL

本文针对实际图像中存在多种类型和程度退化的问题，提出了一种新的多维度调控方法并使用可控残差连接及条件网络构建深度神经网络进行图像修复，实验结果表明该方法在单一和多维度调控任务上均取得了出色表现。