Transfer learning with large pretrained transformer-based language models
like BERT has become a dominating approach for most NLP tasks. Simply
fine-tuning those large language models on downstream tasks or combining it
with task-specific pretraining is often not robust. In particular, the
performance considerably varies as the random seed changes or the number of
pretraining and/or fine-tuning iterations varies, and the fine-tuned model is
vulnerable to adversarial attack. We propose a simple yet effective
adapter-based approach to mitigate these issues. Specifically, we insert small
bottleneck layers (i.e., adapter) within each layer of a pretrained model, then
fix the pretrained layers and train the adapter layers on the downstream task
data, with (1) task-specific unsupervised pretraining and then (2)
task-specific supervised training (e.g., classification, sequence labeling).
Our experiments demonstrate that such a training scheme leads to improved
stability and adversarial robustness in transfer learning to various downstream
tasks.

本文提出了一种简单而有效的适配器方法来提高预训练模型在下游任务的稳定性和抗攻击性。该方法为预训练模型的每层插入小型的瓶颈层，然后在下游任务数据上固定预训练层并训练适配器层，通过在任务特定的无监督训练和监督训练 (e.g., 分类、序列标注) 的过程中，提高了转移学习的性能。