Fine-tuning large pre-trained language models on downstream tasks has become
an important paradigm in NLP. However, common practice fine-tunes all of the
parameters in a pre-trained model, which becomes prohibitive when a large
number of downstream tasks are present. Therefore, many fine-tuning methods are
proposed to learn incremental updates of pre-trained weights in a parameter
efficient way, e.g., low-rank increments. These methods often evenly distribute
the budget of incremental updates across all pre-trained weight matrices, and
overlook the varying importance of different weight parameters. As a
consequence, the fine-tuning performance is suboptimal. To bridge this gap, we
propose AdaLoRA, which adaptively allocates the parameter budget among weight
matrices according to their importance score. In particular, AdaLoRA
parameterizes the incremental updates in the form of singular value
decomposition. Such a novel approach allows us to effectively prune the
singular values of unimportant updates, which is essentially to reduce their
parameter budget but circumvent intensive exact SVD computations. We conduct
extensive experiments with several pre-trained models on natural language
processing, question answering, and natural language generation to validate the
effectiveness of AdaLoRA. Results demonstrate that AdaLoRA manifests notable
improvement over baselines, especially in the low budget settings. Our code is
publicly available at this https URL .

提出 AdaLoRA 方法，以自适应方式在权重矩阵之间分配参数预算，以优化 NLP 的预先训练模型的参数更新问题，在多个自然语言处理任务中表现出显着优势。