Large-scale pre-trained transformers have demonstrated remarkable success in
various computer vision tasks. However, it is still highly challenging to fully
fine-tune these models for downstream tasks due to their high computational and
storage costs. Recently, Parameter-Efficient Tuning (PETuning) techniques,
e.g., Visual Prompt Tuning (VPT) and Low-Rank Adaptation (LoRA), have
significantly reduced the computation and storage cost by inserting lightweight
prompt modules into the pre-trained models and tuning these prompt modules with
a small number of trainable parameters, while keeping the transformer backbone
frozen. Although only a few parameters need to be adjusted, most PETuning
methods still require a significant amount of downstream task training data to
achieve good results. The performance is inadequate on low-data regimes,
especially when there are only one or two examples per class. To this end, we
first empirically identify the poor performance is mainly due to the
inappropriate way of initializing prompt modules, which has also been verified
in the pre-trained language models. Next, we propose a Pre-trained Visual
Parameter-efficient (PVP) Tuning framework, which pre-trains the
parameter-efficient tuning modules first and then leverages the pre-trained
modules along with the pre-trained transformer backbone to perform
parameter-efficient tuning on downstream tasks. Experiment results on five
Fine-Grained Visual Classification (FGVC) and VTAB-1k datasets demonstrate that
our proposed method significantly outperforms state-of-the-art PETuning
methods.

本研究提出了一种基于先前训练模型的视觉快速参数调整 (PVP) 框架，可有效降低由于高计算和存储成本带来的计算和存储成本，并在低数据环境下实现优异的结果，特别是在贫瘠的视觉分类领域中。

PVP：预训练的视觉参数高效微调

PVP: Pre-trained Visual Parameter-Efficient Tuning

Visual Parameter-Efficient Tuning (VPET) has become a powerful alternative
for full fine-tuning so as to adapt pre-trained vision models to downstream
tasks, which only tunes a small number of parameters while freezing the vast
majority ones to ease storage burden and optimization difficulty. However,
existing VPET methods introduce trainable parameters to the same positions
across different tasks depending solely on human heuristics and neglect the
domain gaps. To this end, we study where to introduce and how to allocate
trainable parameters by proposing a novel Sensitivity-aware visual
Parameter-efficient Tuning (SPT) scheme, which adaptively allocates trainable
parameters to task-specific important positions given a desired tunable
parameter budget. Specifically, our SPT first quickly identifies the sensitive
parameters that require tuning for a given task in a data-dependent way. Next,
our SPT further boosts the representational capability for the weight matrices
whose number of sensitive parameters exceeds a pre-defined threshold by
utilizing any of the existing structured tuning methods, e.g., LoRA or Adapter,
to replace directly tuning the selected sensitive parameters (unstructured
tuning) under the budget. Extensive experiments on a wide range of downstream
recognition tasks show that our SPT is complementary to the existing VPET
methods and largely boosts their performance, e.g., SPT improves Adapter with
supervised pre-trained ViT-B/16 backbone by 4.2% and 1.4% mean Top-1 accuracy,
reaching SOTA performance on FGVC and VTAB-1k benchmarks, respectively. Source
code is at this https URL

该文介绍了一种基于感知度的视觉参数调整方案（SPT），该方案可以自适应地将可训练参数分配给特定任务的重要位置，该方法可以有效降低模型存储和优化的难度，并可以取得较好的识别性能。