A primary function of back-propagation is to compute both the gradient of
hidden representations and parameters for optimization with gradient descent.
Training large models requires high computational costs due to their vast
parameter sizes. While Parameter-Efficient Fine-Tuning (PEFT) methods aim to
train smaller auxiliary models to save computational space, they still present
computational overheads, especially in Fine-Tuning as a Service (FTaaS) for
numerous users. We introduce Collaborative Adaptation (ColA) with Gradient
Learning (GL), a parameter-free, model-agnostic fine-tuning approach that
decouples the computation of the gradient of hidden representations and
parameters. In comparison to PEFT methods, ColA facilitates more cost-effective
FTaaS by offloading the computation of the gradient to low-cost devices. We
also provide a theoretical analysis of ColA and experimentally demonstrate that
ColA can perform on par or better than existing PEFT methods on various
benchmarks.

使用协作适应（ColA）和梯度学习（GL）的参数自由、模型不可知的微调方法在性能上与现有的参数高效微调方法相媲美甚至更好，在各种基准测试中 ColA 的计算性能更具成本效益，使得微调作为一项服务可通过将梯度计算卸载到低成本设备来实现。