Parameter-efficient fine-tuning (PEFT) has become increasingly important as
foundation models continue to grow in both popularity and size. Adapter has
been particularly well-received due to their potential for parameter reduction
and adaptability across diverse tasks. However, striking a balance between high
efficiency and robust generalization across tasks remains a challenge for
adapter-based methods. We analyze existing methods and find that: 1) parameter
sharing is the key to reducing redundancy; 2) more tunable parameters, dynamic
allocation, and block-specific design are keys to improving performance.
Unfortunately, no previous work considers all these factors. Inspired by this
insight, we introduce a novel framework named Adapter-X. First, a Sharing
Mixture of Adapters (SMoA) module is proposed to fulfill token-level dynamic
allocation, increased tunable parameters, and inter-block sharing at the same
time. Second, some block-specific designs like Prompt Generator (PG) are
introduced to further enhance the ability of adaptation. Extensive experiments
across 2D image and 3D point cloud modalities demonstrate that Adapter-X
represents a significant milestone as it is the first to outperform full
fine-tuning in both 2D image and 3D point cloud modalities with significantly
fewer parameters, i.e., only 0.20% and 1.88% of original trainable parameters
for 2D and 3D classification tasks. Our code will be publicly available.

Adapter-X 是一种新的框架，通过参数共享和动态分配，以及引入特定设计来提高适应性，在更少的参数下优于传统的微调方法。

Adapter-X: 一个新颖的面向视觉的普适参数高效微调框架

Adapter-X: A Novel General Parameter-Efficient Fine-Tuning Framework for  Vision

Recently, fine-tuning language models pre-trained on large text corpora have
provided huge improvements on vision-and-language (V&L) tasks as well as on
pure language tasks. However, fine-tuning the entire parameter set of
pre-trained models becomes impractical since the model size is growing rapidly.
Hence, in this paper, we introduce adapter-based parameter-efficient transfer
learning techniques to V&L models such as VL-BART and VLT5. We evaluate our
methods in a unified multi-task setup on both image-text and video-text
benchmarks. For the image-text tasks, we use four diverse V&L datasets: VQAv2,
GQA, NLVR2 , and MSCOCO image captioning. For video-text tasks, we use TVQA,
How2QA, TVC, and YC2C. With careful training and thorough experiments, we
benchmark three popular adapter-based methods (Adapter, Hyperformer, Compacter)
against the standard full fine-tuning and the recently proposed prompt-tuning
approach. We also enhance the efficiency and performance of adapters by sharing
their weights to attain knowledge across tasks. Our results demonstrate that
training the adapter with the weight-sharing technique (4.18% of total
parameters for image-text tasks and 3.39% for video-text tasks) can match the
performance of fine-tuning the entire model. Lastly, we present a comprehensive
analysis including the combination of adapter and task-specific prompts and the
impact of V&L pre-training on adapters. Our code is available at:
this https URL

本文提出基于 adapter 的参数高效迁移学习技术，以 VL-BART 和 VLT5 为例，在图像文本和视频文本基准测试上统一多任务设置，通过权重共享提高 adapter 的效率和性能，在图像文本任务和视频文本任务中将 adapter 的使用提升至总参数的 4.18% 和 3.39%，匹配了整个模型微调的性能，同时对 adapter 与任务特定提示的组合及 V&L 预训练对 adapter 的影响进行了综合分析。