It is a common practice in natural language processing to pre-train a single
model on a general domain and then fine-tune it for downstream tasks. However,
when it comes to Large Language Models, fine-tuning the entire model can be
computationally expensive, resulting in very intensive energy consumption. As a
result, several Parameter efficient fine-tuning (PEFT) approaches were recently
proposed. One of the most popular approaches is low-rank adaptation (LoRA),
where the key insight is decomposing the update weights of the pre-trained
model into two low-rank matrices. However, the proposed approaches either use
the same rank value across all different weight matrices or do not use any
quantization technique, which has been shown to be one of the most important
factors when it comes to a model's energy consumption. In this work, we propose
Bayesian-LoRA (B-LoRA) which approaches matrix decomposition and quantization
from a Bayesian perspective by employing a prior distribution on both
quantization levels and rank values of the learned low-rank matrices. As a
result, B-LoRA is able to fine-tune a pre-trained model on a specific
downstream task, finding the optimal rank values and quantization levels for
every low-rank matrix. We validate the proposed model fine-tuning a pre-trained
DeBERTaV3 on the GLUE benchmark. Moreover, we compare it to relevant baselines
and present both qualitative and quantitative results, showing how the proposed
approach is able to learn optimal-rank quantized matrices. B-LoRA performs on
par or better than baselines while reducing the total amount of bit operations
of roughly 70% with respect to the baselines ones.

在这项研究中，我们提出了一种基于贝叶斯角度的矩阵分解和量化方法，名为 B-LoRA，通过对学习到的低秩矩阵的量化级别和秩值引入先验分布，使其能够在特定任务上对预训练模型进行精细调节，找到每个低秩矩阵的最佳秩值和量化级别。通过在 GLUE 基准上对预训练的 DeBERTaV3 模型进行验证，并将其与相关基准进行比较，我们展示了该方法如何学习到最优秩的量化矩阵，相比基线方法，B-LoRA 在性能上与基线相当或更好，并且减少了大约 70% 的总比特操作量。