Fine-tuning large pre-trained language models for downstream tasks remains a
critical challenge in natural language processing. This paper presents an
empirical analysis comparing two efficient fine-tuning methods - BitFit and
adapter modules - to standard full model fine-tuning. Experiments conducted on
GLUE benchmark datasets (MRPC, COLA, STS-B) reveal several key insights. The
BitFit approach, which trains only bias terms and task heads, matches full
fine-tuning performance across varying amounts of training data and time
constraints. It demonstrates remarkable stability even with only 30\% of data,
outperforming full fine-tuning at intermediate data levels. Adapter modules
exhibit high variability, with inconsistent gains over default models. The
findings indicate BitFit offers an attractive balance between performance and
parameter efficiency. Our work provides valuable perspectives on model tuning,
emphasizing robustness and highlighting BitFit as a promising alternative for
resource-constrained or streaming task settings. The analysis offers actionable
guidelines for efficient adaptation of large pre-trained models, while
illustrating open challenges in stabilizing techniques like adapter modules.

为下游任务优化大型预训练语言模型仍然是自然语言处理中的一个关键挑战。本文通过实证分析比较了两种高效微调方法 ——BitFit 和适配器模块与标准的全模型微调。在 GLUE 基准数据集（MRPC、COLA、STS-B）上进行的实验证明了几个关键观点。BitFit 方法仅训练偏置项和任务头，在不同数量的训练数据和时间限制下，与全模型微调相当。即使只使用 30% 的数据，它也表现出明显的稳定性，在中等数据水平上优于全模型微调。适配器模块的性能变异较大，在默认模型上的收益一致性较差。研究结果表明，BitFit 在性能和参数效率之间取得了一个有吸引力的平衡。我们的工作为大型预训练模型的高效调整提供了宝贵的视角，强调了其鲁棒性，并将 BitFit 作为资源受限或流式任务设置的一种有希望的替代方案。该分析提供了有效适应大型预训练模型的可操作指南，同时展示了适配器模块等稳定技术面临的挑战。

大型预训练语言模型的高效微调方法的实证分析

Empirical Analysis of Efficient Fine-Tuning Methods for Large  Pre-Trained Language Models

In this paper, we study the application of Test-Time Training (TTT) as a
solution to handling distribution shifts in speech applications. In particular,
we introduce distribution-shifts to the test datasets of standard
speech-classification tasks -- for example, speaker-identification and
emotion-detection -- and explore how Test-Time Training (TTT) can help adjust
to the distribution-shift. In our experiments that include distribution shifts
due to background noise and natural variations in speech such as gender and
age, we identify some key-challenges with TTT including sensitivity to
optimization hyperparameters (e.g., number of optimization steps and subset of
parameters chosen for TTT) and scalability (e.g., as each example gets its own
set of parameters, TTT is not scalable). Finally, we propose using BitFit -- a
parameter-efficient fine-tuning algorithm proposed for text applications that
only considers the bias parameters for fine-tuning -- as a solution to the
aforementioned challenges and demonstrate that it is consistently more stable
than fine-tuning all the parameters of the model.

本文研究了测试时间训练（TTT）在处理语音应用中的分布偏移问题的应用。我们引入分布偏移到标准语音分类任务的测试数据集中，并探索了测试时间训练（TTT）如何帮助调整到分布偏移。在我们的实验中，包括背景噪声和语音中的性别和年龄等自然变化引起的分布偏移，我们确定了 TTT 面临的一些关键挑战，包括对优化超参数（例如，优化步数和选择 TTT 的参数子集）的敏感性以及可扩展性（例如，由于每个示例都有自己的参数集，TTT 不具有可扩展性）。最后，我们提出使用 BitFit 作为解决上述挑战的参数高效微调算法，BitFit 仅考虑偏差参数进行微调，并证明它相对于微调模型的所有参数更加稳定。

测试时训练语音

Test-Time Training for Speech

We introduce BitFit, a sparse-finetuning method where only the bias-terms of
the model (or a subset of them) are being modified. We show that with
small-to-medium training data, applying BitFit on pre-trained BERT models is
competitive with (and sometimes better than) fine-tuning the entire model. For
larger data, the method is competitive with other sparse fine-tuning methods.
Besides their practical utility, these findings are relevant for the question
of understanding the commonly-used process of finetuning: they support the
hypothesis that finetuning is mainly about exposing knowledge induced by
language-modeling training, rather than learning new task-specific linguistic
knowledge.

介绍了 BitFit 方法，该方法在模型的偏置项（或其中的子集）被修改时进行了稀疏微调。使用小到中等规模的训练数据，将 BitFit 应用于预训练的 BERT 模型与整个模型微调相比具有竞争力（有时甚至更优）。对于更大的数据，该方法与其他稀疏微调方法具有竞争力。此外，这些发现与理解微调的常用过程的问题相关，他们支持这样的假设：微调主要是关于展示由语言建模训练产生的知识，而不是学习新的任务特定的语言知识。