Popular parameter-efficient fine-tuning (PEFT) methods, such as LoRA and its
variants, freeze pre-trained model weights \(W\) and inject learnable matrices
\(\Delta W\). These \(\Delta W\) matrices are structured for efficient
parameterization, often using techniques like low-rank approximations or
scaling vectors. However, these methods typically show a performance gap
compared to full fine-tuning. Although recent PEFT methods have narrowed this
gap, they do so at the cost of additional learnable parameters. We propose
SVFT, a simple approach that fundamentally differs from existing methods: the
structure imposed on \(\Delta W\) depends on the specific weight matrix \(W\).
Specifically, SVFT updates \(W\) as a sparse combination of outer products of
its singular vectors, training only the coefficients (scales) of these sparse
combinations. This approach allows fine-grained control over expressivity
through the number of coefficients. Extensive experiments on language and
vision benchmarks show that SVFT recovers up to 96% of full fine-tuning
performance while training only 0.006 to 0.25% of parameters, outperforming
existing methods that only recover up to 85% performance using 0.03 to 0.8% of
the trainable parameter budget.

通过在参数更新中使用矩阵的稀疏组合，SVFT 方法在仅使用 0.006% 到 0.25% 的可训练参数时，能够恢复高达 96% 的全面微调性能，超过了仅使用 0.03% 到 0.8% 的可训练参数预算恢复的最高 85% 性能。