The increasing scale of vision transformers (ViT) has made the efficient
fine-tuning of these large models for specific needs a significant challenge in
various applications. This issue originates from the computationally demanding
matrix multiplications required during the backpropagation process through
linear layers in ViT. In this paper, we tackle this problem by proposing a new
Low-rank BackPropagation via Walsh-Hadamard Transformation (LBP-WHT) method.
Intuitively, LBP-WHT projects the gradient into a low-rank space and carries
out backpropagation. This approach substantially reduces the computation needed
for adapting ViT, as matrix multiplication in the low-rank space is far less
resource-intensive. We conduct extensive experiments with different models
(ViT, hybrid convolution-ViT model) on multiple datasets to demonstrate the
effectiveness of our method. For instance, when adapting an EfficientFormer-L1
model on CIFAR100, our LBP-WHT achieves 10.4% higher accuracy than the
state-of-the-art baseline, while requiring 9 MFLOPs less computation. As the
first work to accelerate ViT adaptation with low-rank backpropagation, our
LBP-WHT method is complementary to many prior efforts and can be combined with
them for better performance.

提出了一种名为 LBP-WHT 的新方法来加速 Vision Transformers（ViT）的 Fine-tuning 过程，通过将梯度投影到低秩空间来显著减少计算需求，实验证明该方法在多个数据集上有效，并与其他方法相结合以取得更好性能。