Large pretrained language models (PLMs) are often domain- or task-adapted via
fine-tuning or prompting. Finetuning requires modifying all of the parameters
and having enough data to avoid overfitting while prompting requires no
training and few examples but limits performance. Instead, we prepare PLMs for
data- and parameter-efficient adaptation by learning to learn the difference
between general and adapted PLMs. This difference is expressed in terms of
model weights and sublayer structure through our proposed dynamic low-rank
reparameterization and learned architecture controller. Experiments on few-shot
dialogue completion, low-resource abstractive summarization, and multi-domain
language modeling show improvements in adaptation time and performance over
direct finetuning or preparation via domain-adaptive pretraining. Ablations
show our task-adaptive reparameterization (TARP) and model search (TAMS)
components individually improve on other parameter-efficient transfer like
adapters and structure-learning methods like learned sparsification.

该论文提出了一种通过学习区分微调和适应性 PLMs 的方法，使用动态低秩重参数化和学习结构控制器来实现数据和参数高效的自适应，并在对话完成、摘要生成等任务上展示出明显提升。