State-of-the-art pretrained NLP models contain a hundred million to trillion
parameters. Adapters provide a parameter-efficient alternative for the full
finetuning in which we can only finetune lightweight neural network layers on
top of pretrained weights. Adapter layers are initialized randomly. However,
existing work uses the same adapter architecture -- i.e., the same adapter
layer on top of each layer of the pretrained model -- for every dataset,
regardless of the properties of the dataset or the amount of available training
data. In this work, we introduce adaptable adapters that contain (1) learning
different activation functions for different layers and different input data,
and (2) a learnable switch to select and only use the beneficial adapter
layers. We show that adaptable adapters achieve on-par performances with the
standard adapter architecture while using a considerably smaller number of
adapter layers. In addition, we show that the selected adapter architecture by
adaptable adapters transfers well across different data settings and similar
tasks. We propose to use adaptable adapters for designing efficient and
effective adapter architectures. The resulting adapters (a) contain about 50%
of the learning parameters of the standard adapter and are therefore more
efficient at training and inference, and require less storage space, and (b)
achieve considerably higher performances in low-data settings.

本文提出了一种适应能力强的适配器层，不同数据集可采用不同的激活函数及适配器层，并且可选择最优的适配器层，以达到与标准适配器层相近的性能表现。这种新型适配器层可使模型的存储及训练效率更高，并能在低数据情况下取得更好的性能表现。