Recent pretrained language models extend from millions to billions of
parameters. Thus the need to fine-tune an extremely large pretrained model with
a limited training corpus arises in various downstream tasks. In this paper, we
propose a straightforward yet effective fine-tuning technique, Child-Tuning,
which updates a subset of parameters (called child network) of large pretrained
models via strategically masking out the gradients of the non-child network
during the backward process. Experiments on various downstream tasks in GLUE
benchmark show that Child-Tuning consistently outperforms the vanilla
fine-tuning by 1.5~8.6 average score among four different pretrained models,
and surpasses the prior fine-tuning techniques by 0.6~1.3 points. Furthermore,
empirical results on domain transfer and task transfer show that Child-Tuning
can obtain better generalization performance by large margins.

本篇文章提出了名为 Child-Tuning 的 fine-tuning 技术，通过在反向传播过程中遮盖非子网络的梯度来更新大型预训练模型的子网络的子集，实验结果表明 Child-Tuning 在 GLUE 基准测试的各个下游任务中始终优于普通 fine-tuning，平均分数高出 1.5~8.6 分。此外，领域转移和任务转移的实证结果表明，Child-Tuning 可以大幅提高泛化性能。