Pre-training & fine-tuning is a prevalent paradigm in computer vision (CV).
Recently, parameter-efficient transfer learning (PETL) methods have shown
promising performance in transferring knowledge from pre-trained models with
only a few trainable parameters. Despite their success, the existing PETL
methods in CV can be computationally expensive and require large amounts of
memory and time cost during training, which limits low-resource users from
conducting research and applications on large models. In this work, we propose
Parameter, Memory, and Time Efficient Visual Adapter ($\mathrm{E^3VA}$) tuning
to address this issue. We provide a gradient backpropagation highway for
low-rank adapters which removes large gradient computations for the frozen
pre-trained parameters, resulting in substantial savings of training memory and
training time. Furthermore, we optimise the $\mathrm{E^3VA}$ structure for
dense predictions tasks to promote model performance. Extensive experiments on
COCO, ADE20K, and Pascal VOC benchmarks show that $\mathrm{E^3VA}$ can save up
to 62.2% training memory and 26.2% training time on average, while achieving
comparable performance to full fine-tuning and better performance than most
PETL methods. Note that we can even train the Swin-Large-based Cascade Mask
RCNN on GTX 1080Ti GPUs with less than 1.5% trainable parameters.

本文提出了一种名为 E3VA 的参数、内存和时间高效的视觉适配器调节方法，通过梯度反向传播高速公路实现低秩适配器，可以在保持模型性能相当的情况下，节省高达 62.2% 的训练存储器和 26.2% 的训练时间。