A major factor in the recent success of large language models is the use of
enormous and ever-growing text datasets for unsupervised pre-training. However,
naively training a model on all available data may not be optimal (or
feasible), as the quality of available text data can vary. Filtering out data
can also decrease the carbon footprint and financial costs of training models
by reducing the amount of training required.
Data selection methods aim to determine which candidate data points to
include in the training dataset and how to appropriately sample from the
selected data points. The promise of improved data selection methods has caused
the volume of research in the area to rapidly expand. However, because deep
learning is mostly driven by empirical evidence and experimentation on
large-scale data is expensive, few organizations have the resources for
extensive data selection research. Consequently, knowledge of effective data
selection practices has become concentrated within a few organizations, many of
which do not openly share their findings and methodologies.
To narrow this gap in knowledge, we present a comprehensive review of
existing literature on data selection methods and related research areas,
providing a taxonomy of existing approaches. By describing the current
landscape of research, this work aims to accelerate progress in data selection
by establishing an entry point for new and established researchers.
Additionally, throughout this review we draw attention to noticeable holes in
the literature and conclude the paper by proposing promising avenues for future
research.

该论文综述了大型语言模型、数据选择方法、经验证据、大规模数据选择研究和未来研究方向的相关领域，旨在为新老研究人员提供入门点，加速数据选择领域的进展。

语言模型数据选择调查

A Survey on Data Selection for Language Models

Instruction tuning is a vital step of training large language models (LLM),
so how to enhance the effect of instruction tuning has received increased
attention. Existing works indicate that the quality of the dataset is more
crucial than the quantity during instruction tuning of LLM. Therefore, recently
a lot of studies focus on exploring the methods of selecting high-quality
subset from instruction datasets, aiming to reduce training costs and enhance
the instruction-following capabilities of LLMs. This paper presents a
comprehensive survey on data selection for LLM instruction tuning. Firstly, we
introduce the wildly used instruction datasets. Then, we propose a new taxonomy
of the data selection methods and provide a detailed introduction of recent
advances,and the evaluation strategies and results of data selection methods
are also elaborated in detail. Finally, we emphasize the open challenges and
present new frontiers of this task.

通过介绍使用的指导数据集，提出了数据选择方法的新分类，并详细介绍了最近的研究进展、评估策略和结果，强调了这一任务的挑战和新的前沿。

LLM 指导调优的数据选择调查

A Survey on Data Selection for LLM Instruction Tuning

The data used to pretrain large language models has a decisive impact on a
model's downstream performance, which has led to a large body of work on data
selection methods that aim to automatically determine the most suitable data to
use for pretraining. Existing data selection methods suffer from slow and
computationally expensive processes, a problem amplified by the increasing size
of models and of pretraining datasets. Data mixing, on the other hand, reduces
the complexity of data selection by grouping data points together and
determining sampling probabilities across entire groups. However, data mixing
proportions are typically fixed before training and therefore cannot adapt to
changing training dynamics. To address these limitations, we develop an
efficient algorithm for Online Data Mixing (ODM) that combines elements from
both data selection and data mixing. Based on multi-armed bandit algorithms,
our online approach optimizes the data mixing proportions during training.
Remarkably, our method trains a model that reaches the final perplexity of the
next best method with 19\% fewer training iterations, and improves performance
on the 5-shot MMLU benchmark by 1.9% relative accuracy, while adding negligible
wall-clock time during pretraining.

研究论文通过开发一种高效的在线数据混合算法（ODM），结合了数据选择和数据混合的元素，优化了训练过程中的数据混合比例，提高了大型语言模型的性能。

语言模型预训练的高效在线数据混合

Efficient Online Data Mixing For Language Model Pre-Training

While data selection methods have been studied extensively in active
learning, data pruning, and data augmentation settings, there is little
evidence for the efficacy of these methods in industry scale settings,
particularly in low-resource languages. Our work presents ways of assessing
prospective training examples in those settings for their "usefulness" or
"difficulty". We also demonstrate how these measures can be used in selecting
important examples for training supervised machine learning models. We
primarily experiment with entropy and Error L2-Norm (EL2N) scores. We use these
metrics to curate high quality datasets from a large pool of \textit{Weak
Signal Labeled} data, which assigns no-defect high confidence hypotheses during
inference as ground truth labels. We then conduct training data augmentation
experiments using these de-identified datasets and demonstrate that score-based
selection can result in a 2% decrease in semantic error rate and 4%-7% decrease
in domain classification error rate when compared to the baseline technique of
random selection.

在工业规模的环境中，特别是在资源有限的语言中，我们通过熵和 EL2N 评分来评估潜在的训练样本的 “有用性” 或 “难度”，并展示了如何使用这些度量来选择用于训练监督机器学习模型的重要样本。我们使用这些度量从大量的 “弱信号标记” 数据中筛选高质量的数据集，然后进行以评分为基础的选择来增强训练数据实验，与随机选择的基线技术相比，结果显示了语义错误率下降 2% 和领域分类错误率下降 4%-7%。