Adapting large-scale pre-trained generative models in a parameter-efficient
manner is gaining traction. Traditional methods like low rank adaptation
achieve parameter efficiency by imposing constraints but may not be optimal for
tasks requiring high representation capacity. We propose a novel spectrum-aware
adaptation framework for generative models. Our method adjusts both singular
values and their basis vectors of pretrained weights. Using the Kronecker
product and efficient Stiefel optimizers, we achieve parameter-efficient
adaptation of orthogonal matrices. We introduce Spectral Orthogonal
Decomposition Adaptation (SODA), which balances computational efficiency and
representation capacity. Extensive evaluations on text-to-image diffusion
models demonstrate SODA's effectiveness, offering a spectrum-aware alternative
to existing fine-tuning methods.

提出了一种新的谱感知适应框架，即 SODA，用于大规模预训练生成模型的参数高效适应，通过调整预训练权重的奇异值和基向量来实现参数高效适应正交矩阵，提供了一种谱感知替代现有微调方法的有效选择。