Adapter Tuning, which freezes the pretrained language models (PLMs) and only
fine-tunes a few extra modules, becomes an appealing efficient alternative to
the full model fine-tuning. Although computationally efficient, the recent
Adapters often increase parameters (e.g. bottleneck dimension) for matching the
performance of full model fine-tuning, which we argue goes against their
original intention. In this work, we re-examine the parameter-efficiency of
Adapters through the lens of network pruning (we name such plug-in concept as
\texttt{SparseAdapter}) and find that SparseAdapter can achieve comparable or
better performance than standard Adapters when the sparse ratio reaches up to
80\%. Based on our findings, we introduce an easy but effective setting
``\textit{Large-Sparse}'' to improve the model capacity of Adapters under the
same parameter budget. Experiments on five competitive Adapters upon three
advanced PLMs show that with proper sparse method (e.g. SNIP) and ratio (e.g.
40\%) SparseAdapter can consistently outperform their corresponding
counterpart. Encouragingly, with the \textit{Large-Sparse} setting, we can
obtain further appealing gains, even outperforming the full fine-tuning by a
large margin. Our code will be released at:
this https URL.

插件调整适用于预训练语言模型中；本研究基于稀疏适配器通过带放大系数的参数情况，实现了大的模型容量；通过使用 SNIP 方法和适当的系数，稀疏适配器可以始终优于对应的常规适配器，并且大稀疏设置可以获得进一步的性能提升，甚至可以大幅超过完全微调的性能。