This work focuses on leveraging and selecting from vast, unlabeled, open data
to pre-fine-tune a pre-trained language model. The goal is to minimize the need
for costly domain-specific data for subsequent fine-tuning while achieving
desired performance levels. While many data selection algorithms have been
designed for small-scale applications, rendering them unsuitable for our
context, some emerging methods do cater to language data scales. However, they
often prioritize data that aligns with the target distribution. While this
strategy may be effective when training a model from scratch, it can yield
limited results when the model has already been pre-trained on a different
distribution. Differing from prior work, our key idea is to select data that
nudges the pre-training distribution closer to the target distribution. We show
the optimality of this approach for fine-tuning tasks under certain conditions.
We demonstrate the efficacy of our methodology across a diverse array of tasks
(NLU, NLG, zero-shot) with models up to 2.7B, showing that it consistently
surpasses other selection methods. Moreover, our proposed method is
significantly faster than existing techniques, scaling to millions of samples
within a single GPU hour. Our code is open-sourced (Code repository:
this https URL ). While fine-tuning offers
significant potential for enhancing performance across diverse tasks, its
associated costs often limit its widespread adoption; with this work, we hope
to lay the groundwork for cost-effective fine-tuning, making its benefits more
accessible.

使用无标签开放数据进行预精调预训练语言模型以最小化领域特定数据需求，并在达到预期性能水平的同时选择数据以将预训练分布推向目标分布的优化方法，展示了该方法在各种任务中的优越性和快速性，为经济高效的微调奠定了基础。

從原則出發的數據選擇，為 LLMs 的預熱微調提供更多、花費更少

Get more for less: Principled Data Selection for Warming Up Fine-Tuning  in LLMs

Recent progress in Automatic Speech Recognition (ASR) has been coupled with a
substantial increase in the model sizes, which may now contain billions of
parameters, leading to slow inferences even with adapted hardware. In this
context, several ASR models exist in various sizes, with different inference
costs leading to different performance levels. Based on the observation that
smaller models perform optimally on large parts of testing corpora, we propose
to train a decision module, that would allow, given an audio sample, to use the
smallest sufficient model leading to a good transcription. We apply our
approach to two Whisper models with different sizes. By keeping the decision
process computationally efficient, we build a decision module that allows
substantial computational savings with reduced performance drops.

基于自动语音识别（ASR）领域中模型尺寸和推理成本的观察，通过训练一个决策模块，可以在测试数据的大部分模块上使用最小的足够模型达到良好的转录效果，从而实现了相当大的计算节省和性能提升。