Parameter-efficient fine-tuning (PEFT) has emerged as a popular approach for
adapting pre-trained Vision Transformer (ViT) models to downstream
applications. While current PEFT methods achieve parameter efficiency, they
overlook GPU memory and time efficiency during both fine-tuning and inference,
due to the repeated computation of redundant tokens in the ViT architecture.
This falls short of practical requirements for downstream task adaptation. In
this paper, we propose \textbf{Sparse-Tuning}, a novel tuning paradigm that
substantially enhances both fine-tuning and inference efficiency for
pre-trained ViT models. Sparse-Tuning efficiently fine-tunes the pre-trained
ViT by sparsely preserving the informative tokens and merging redundant ones,
enabling the ViT to focus on the foreground while reducing computational costs
on background regions in the images. To accurately distinguish informative
tokens from uninformative ones, we introduce a tailored Dense Adapter, which
establishes dense connections across different encoder layers in the ViT,
thereby enhancing the representational capacity and quality of token
sparsification. Empirical results on VTAB-1K, three complete image datasets,
and two complete video datasets demonstrate that Sparse-Tuning reduces the
GFLOPs to \textbf{62\%-70\%} of the original ViT-B while achieving
state-of-the-art performance. Source code is available at
https://github.com/liuting20/Sparse-Tuning.

Sparse-Tuning 是一种新的调优范式，通过稀疏保存信息标记并合并冗余标记，提高对前景的关注并降低背景区域的计算成本，实现了对预训练的 ViT 模型进行高效的微调和推断，同时具备了现有方法无法满足的 GPU 内存和时间效率要求。