Adapter-Tuning (AT) method involves freezing a pre-trained model and
introducing trainable adapter modules to acquire downstream knowledge, thereby
calibrating the model for better adaptation to downstream tasks. This paper
proposes a distillation framework for the AT method instead of crafting a
carefully designed adapter module, which aims to improve fine-tuning
performance. For the first time, we explore the possibility of combining the AT
method with knowledge distillation. Via statistical analysis, we observe
significant differences in the knowledge acquisition between adapter modules of
different models. Leveraging these differences, we propose a simple yet
effective framework called inverse Distillation Adapter-Tuning (iDAT).
Specifically, we designate the smaller model as the teacher and the larger
model as the student. The two are jointly trained, and online knowledge
distillation is applied to inject knowledge of different perspective to student
model, and significantly enhance the fine-tuning performance on downstream
tasks. Extensive experiments on the VTAB-1K benchmark with 19 image
classification tasks demonstrate the effectiveness of iDAT. The results show
that using existing AT method within our iDAT framework can further yield a
2.66% performance gain, with only an additional 0.07M trainable parameters. Our
approach compares favorably with state-of-the-arts without bells and whistles.
Our code is available at this https URL

通过利用逆向蒸馏适配器调优（iDAT）框架，将 Adapter-Tuning 方法和知识蒸馏相结合，我们成功地改进了微调性能，并实验证明了该方法的有效性。