Current parameter-efficient fine-tuning (PEFT) methods build adapters without
considering the context of downstream task to learn, or the context of
important knowledge to maintain. As a result, there is often a performance gap
compared to full-parameter finetuning, and meanwhile the finetuned model
suffers from catastrophic forgetting of the pre-trained world knowledge. In
this paper, we propose CorDA, a Context-oriented Decomposition Adaptation
method that builds learnable adapters from weight decomposition oriented by the
context of downstream task or world knowledge. Concretely, we collect a few
data samples, and perform singular value decomposition for each linear layer of
a pre-trained LLM multiplied by the covariance matrix of the input activation
using these samples. By doing so, the context of the representative samples is
captured through deciding the factorizing orientation. Our method enables two
options, the knowledge-preserved adaptation and the instruction-previewed
adaptation. For the former, we use question-answering samples to obtain the
covariance matrices, and use the decomposed components with the smallest $r$
singular values to initialize a learnable adapter, with the others frozen such
that the world knowledge is better preserved. For the latter, we use the
instruction data from the finetuning task, such as math or coding, to orientate
the decomposition and train the largest $r$ components that capture the main
characteristics of the task to learn. We conduct extensive experiments on Math,
Code, and Instruction Following tasks. Our knowledge-preserved adaptation not
only achieves better performance than LoRA on finetuning tasks, but also
mitigates the forgetting of world knowledge. Our instruction-previewed
adaptation is able to further enhance the finetuning performance, surpassing
full-parameter finetuning and the state-of-the-art PEFT methods.

该研究论文提出了一种基于上下文的分解自适应方法 CorDA，通过收集少量数据样本并使用奇异值分解来构建适应器，从而更好地保留了世界知识并提升了参数优化方法的性能。