Adapter-Tuning (AT) method involves freezing a pre-trained model and
introducing trainable adapter modules to acquire downstream knowledge, thereby
calibrating the model for better adaptation to downstream tasks. This paper
proposes a distillation framework for the AT method instead of crafting a
carefully designed adapter module, which aims to improve fine-tuning
performance. For the first time, we explore the possibility of combining the AT
method with knowledge distillation. Via statistical analysis, we observe
significant differences in the knowledge acquisition between adapter modules of
different models. Leveraging these differences, we propose a simple yet
effective framework called inverse Distillation Adapter-Tuning (iDAT).
Specifically, we designate the smaller model as the teacher and the larger
model as the student. The two are jointly trained, and online knowledge
distillation is applied to inject knowledge of different perspective to student
model, and significantly enhance the fine-tuning performance on downstream
tasks. Extensive experiments on the VTAB-1K benchmark with 19 image
classification tasks demonstrate the effectiveness of iDAT. The results show
that using existing AT method within our iDAT framework can further yield a
2.66% performance gain, with only an additional 0.07M trainable parameters. Our
approach compares favorably with state-of-the-arts without bells and whistles.
Our code is available at this https URL

通过利用逆向蒸馏适配器调优（iDAT）框架，将 Adapter-Tuning 方法和知识蒸馏相结合，我们成功地改进了微调性能，并实验证明了该方法的有效性。

iDAT: 逆蒸馏适配器调整

iDAT: inverse Distillation Adapter-Tuning

Parameter-efficient fine-tuning (PEFT) has shown its effectiveness in
adapting the pre-trained language models to downstream tasks while only
updating a small number of parameters. Despite the success, most existing
methods independently adapt to each task without considering knowledge transfer
between tasks and are limited to low-data regimes. To overcome this issue, we
propose Prototype-based HyperAdapter (PHA), a novel framework built on the
adapter-tuning and hypernetwork. It introduces an instance-dense retriever and
a prototypical hypernetwork to generate the conditional modules in a
sample-efficient manner. This leads to comparable performance improvements
against existing PEFT methods on multi-task learning and few-shot transfer
learning. More importantly, when the available data size gets smaller, our
method outperforms other strong baselines by a large margin. Based on our
extensive empirical experiments across various datasets, we demonstrate that
PHA strikes a better trade-off between trainable parameters, accuracy on stream
tasks, and sample efficiency.

基于 Protoype 的 HyperAdapter（PHA）是一个建立在 adapter-tuning 和超网络的新框架，它采用实例密集检索器和原型超网络以更有效地生成条件模块，从而在多任务学习和少样本迁移学习中与现有的参数有效微调（PEFT）方法相比具有可比较的性能改进，并在数据规模较小时，与其他强基准相比，取得了显著的优势。基于对各种数据集的大量实证实验，我们证明 PHA 在可训练参数、模型精度和样本效率之间取得了更好的平衡。

基于原型的超适配器用于高效的多任务调优

Prototype-based HyperAdapter for Sample-Efficient Multi-task Tuning

Adapter-tuning is a paradigm that transfers a pretrained language model to
downstream tasks by adding and tuning a small number of new parameters.
Previously proposed adapter architectures are all feed-forward neural networks.
In this paper, we investigate the effectiveness of using tiny-attention --
i.e., attention with extremely small per-head dimensionality -- as adapters.
Our tiny-attention adapter learns to modify the hidden states at each position
directly conditioned on the hidden states at all the other positions, which is
missed by the previously proposed adapters. Moreover, we view its multiple
attention heads as a mixture of experts and propose to average their weights
during deployment, which further reduces its inference computation cost. On the
GLUE benchmark, our tiny-attention adapter outperforms the other
parameter-efficient transfer learning methods as well as full fine-tuning while
only updating 0.05% of the parameters. On the FewGLUE benchmark, its
performance is comparable to that of GPT-3 and PET.

本文探讨了使用小型注意力机制作为适配器的有效性及其在 GLUE 基准测试和 FewGLUE 基准测试方面的性能表现，证明了相比于以前的方法，它能针对下游任务进行更好的参数调整和更新。