We introduce BitFit, a sparse-finetuning method where only the bias-terms of
the model (or a subset of them) are being modified. We show that with
small-to-medium training data, applying BitFit on pre-trained BERT models is
competitive with (and sometimes better than) fine-tuning the entire model. For
larger data, the method is competitive with other sparse fine-tuning methods.
Besides their practical utility, these findings are relevant for the question
of understanding the commonly-used process of finetuning: they support the
hypothesis that finetuning is mainly about exposing knowledge induced by
language-modeling training, rather than learning new task-specific linguistic
knowledge.

介绍了 BitFit 方法，该方法在模型的偏置项（或其中的子集）被修改时进行了稀疏微调。使用小到中等规模的训练数据，将 BitFit 应用于预训练的 BERT 模型与整个模型微调相比具有竞争力（有时甚至更优）。对于更大的数据，该方法与其他稀疏微调方法具有竞争力。此外，这些发现与理解微调的常用过程的问题相关，他们支持这样的假设：微调主要是关于展示由语言建模训练产生的知识，而不是学习新的任务特定的语言知识。