Recent work has explored the potential to adapt a pre-trained vision
transformer (ViT) by updating only a few parameters so as to improve storage
efficiency, called parameter-efficient transfer learning (PETL). Current PETL
methods have shown that by tuning only 0.5% of the parameters, ViT can be
adapted to downstream tasks with even better performance than full fine-tuning.
In this paper, we aim to further promote the efficiency of PETL to meet the
extreme storage constraint in real-world applications. To this end, we propose
a tensorization-decomposition framework to store the weight increments, in
which the weights of each ViT are tensorized into a single 3D tensor, and their
increments are then decomposed into lightweight factors. In the fine-tuning
process, only the factors need to be updated and stored, termed Factor-Tuning
(FacT). On VTAB-1K benchmark, our method performs on par with NOAH, the
state-of-the-art PETL method, while being 5x more parameter-efficient. We also
present a tiny version that only uses 8K (0.01% of ViT's parameters) trainable
parameters but outperforms full fine-tuning and many other PETL methods such as
VPT and BitFit. In few-shot settings, FacT also beats all PETL baselines using
the fewest parameters, demonstrating its strong capability in the low-data
regime.

本研究针对实际应用中的存储压力，提出了一种张量分解框架，称为 Factor-Tuning（FacT），通过将 ViT 的权重张量化为一个 3D 张量，并将它们的增量分解为轻量级因子，从而实现 PETL 的极限存储效率，实验结果表明，FacT 在少量可训练参数（0.01%）的情况下，表现优于全微调和许多其他 PETL 方法，并在少样本学习中显示其强大的能力.