Visual Parameter-Efficient Tuning (VPET) has become a powerful alternative
for full fine-tuning so as to adapt pre-trained vision models to downstream
tasks, which only tunes a small number of parameters while freezing the vast
majority ones to ease storage burden and optimization difficulty. However,
existing VPET methods introduce trainable parameters to the same positions
across different tasks depending solely on human heuristics and neglect the
domain gaps. To this end, we study where to introduce and how to allocate
trainable parameters by proposing a novel Sensitivity-aware visual
Parameter-efficient Tuning (SPT) scheme, which adaptively allocates trainable
parameters to task-specific important positions given a desired tunable
parameter budget. Specifically, our SPT first quickly identifies the sensitive
parameters that require tuning for a given task in a data-dependent way. Next,
our SPT further boosts the representational capability for the weight matrices
whose number of sensitive parameters exceeds a pre-defined threshold by
utilizing any of the existing structured tuning methods, e.g., LoRA or Adapter,
to replace directly tuning the selected sensitive parameters (unstructured
tuning) under the budget. Extensive experiments on a wide range of downstream
recognition tasks show that our SPT is complementary to the existing VPET
methods and largely boosts their performance, e.g., SPT improves Adapter with
supervised pre-trained ViT-B/16 backbone by 4.2% and 1.4% mean Top-1 accuracy,
reaching SOTA performance on FGVC and VTAB-1k benchmarks, respectively. Source
code is at this https URL

该文介绍了一种基于感知度的视觉参数调整方案（SPT），该方案可以自适应地将可训练参数分配给特定任务的重要位置，该方法可以有效降低模型存储和优化的难度，并可以取得较好的识别性能。