Finetuning a pretrained model has become a standard approach for training
neural networks on novel tasks, resulting in fast convergence and improved
performance. In this work, we study an alternative finetuning method, where
instead of finetuning all the weights of the network, we only train a carefully
chosen subset of layers, keeping the rest of the weights frozen at their
initial (pretrained) values. We demonstrate that \emph{subset finetuning} (or
SubTuning) often achieves accuracy comparable to full finetuning of the model,
and even surpasses the performance of full finetuning when training data is
scarce. Therefore, SubTuning allows deploying new tasks at minimal
computational cost, while enjoying the benefits of finetuning the entire model.
This yields a simple and effective method for multi-task learning, where
different tasks do not interfere with one another, and yet share most of the
resources at inference time. We demonstrate the efficiency of SubTuning across
multiple tasks, using different network architectures and pretraining methods.

本研究探讨了一种新的用于 fine-tuning 神经网络的方法，称为子集 fine-tuning，即只对部分层进行精细调整并锁定其余权重，该方法可以在不增加计算成本的情况下实现多任务的学习和推理，并在数据稀缺的情况下实现与完全 fine-tuning 模型相当甚至更好的性能。