While task-specific finetuning of pretrained networks has led to significant empirical advances in NLP, the large size of networks makes finetuning difficult to deploy in multi-task, memory-constrained settings. We propose diff pruning as a simple approach to enable parameter-efficient transfer learning within the pretrain-finetune framework. This approach views finetuning as learning a task-specific diff vector that is applied on top of the pretrained parameter vector, which remains fixed and is shared across different tasks. The diff vector is adaptively pruned during training with a differentiable approximation to the L0-norm penalty to encourage sparsity. Diff pruning becomes parameter-efficient as the number of tasks increases, as it requires storing only the nonzero positions and weights of the diff vector for each task, while the cost of storing the shared pretrained model remains constant. It further does not require access to all tasks during training, which makes it attractive in settings where tasks arrive in stream or the set of tasks is unknown. We find that models finetuned with diff pruning can match the performance of fully finetuned baselines on the GLUE benchmark while only modifying 0.5% of the pretrained model's parameters per task.

提出一种在预训练-微调框架下进行参数有效转移学习的简单方法 - 差分裁剪（diff pruning），它将微调视为学习一种针对特定任务的“diff vector”，该向量附加在预训练的参数向量之上。通过将不同iable逼近L0-norm 惩罚来自适应地修剪diff向量以鼓励稀疏性，从而在学习时适应地进行修剪，从而在与完全微调的基准模型相比性能匹配，并且每项任务仅修改预训练模型参数的0.5%。

Diff剪枝实现参数高效的迁移学习