The pretrain-then-finetune paradigm has been widely adopted in computer
vision. But as the size of Vision Transformer (ViT) grows exponentially, the
full finetuning becomes prohibitive in view of the heavier storage overhead.
Motivated by parameter-efficient transfer learning (PETL) on language
transformers, recent studies attempt to insert lightweight adaptation modules
(e.g., adapter layers or prompt tokens) to pretrained ViT and only finetune
these modules while the pretrained weights are frozen. However, these modules
were originally proposed to finetune language models and did not take into
account the prior knowledge specifically for visual tasks. In this paper, we
propose to construct Convolutional Bypasses (Convpass) in ViT as adaptation
modules, introducing only a small amount (less than 0.5% of model parameters)
of trainable parameters to adapt the large ViT. Different from other PETL
methods, Convpass benefits from the hard-coded inductive bias of convolutional
layers and thus is more suitable for visual tasks, especially in the low-data
regime. Experimental results on VTAB-1K benchmark and few-shot learning
datasets show that Convpass outperforms current language-oriented adaptation
modules, demonstrating the necessity to tailor vision-oriented adaptation
modules for adapting vision models.

本文提出使用卷积旁路（Convpass）在预训练的 Vision Transformer 中作为适应模块，这个方法只需要少量可训练参数即可适应大型 Vision Transformer，其表现优于当前的适应模块，并证明为了适应视觉模型需要定制视觉定向的适应模块。

卷积旁路更好的视觉 Transformer 适配器

Convolutional Bypasses Are Better Vision Transformer Adapters

Standard fine-tuning of large pre-trained language models (PLMs) for
downstream tasks requires updating hundreds of millions to billions of
parameters, and storing a large copy of the PLM weights for every task
resulting in increased cost for storing, sharing and serving the models. To
address this, parameter-efficient fine-tuning (PEFT) techniques were introduced
where small trainable components are injected in the PLM and updated during
fine-tuning. We propose AdaMix as a general PEFT method that tunes a mixture of
adaptation modules -- given the underlying PEFT method of choice -- introduced
in each Transformer layer while keeping most of the PLM weights frozen. For
instance, AdaMix can leverage a mixture of adapters like Houlsby or a mixture
of low rank decomposition matrices like LoRA to improve downstream task
performance over the corresponding PEFT methods for fully supervised and
few-shot NLU and NLG tasks. Further, we design AdaMix such that it matches the
same computational cost and the number of tunable parameters as the underlying
PEFT method. By only tuning 0.1-0.2% of PLM parameters, we show that AdaMix
outperforms SOTA parameter-efficient fine-tuning and full model fine-tuning for
both NLU and NLG tasks.

本文提出 AdaMix 作为一种通用的参数高效微调方法，通过调整每个 Transformer 层中引入的自适应模块的混合来提高下游自然语言理解和生成任务的性能。这种方法只对 0.1-0.2％的 PLM 参数进行微调，可以超越目前的各种参数高效微调和完整模型微调方法。