Fine-tuning large-scale pre-trained models is inherently a resource-intensive
task. While it can enhance the capabilities of the model, it also incurs
substantial computational costs, posing challenges to the practical application
of downstream tasks. Existing parameter-efficient fine-tuning (PEFT) methods
such as Low-Rank Adaptation (LoRA) rely on a bypass framework that ignores the
differential parameter budget requirements across weight matrices, which may
lead to suboptimal fine-tuning outcomes. To address this issue, we introduce
the Dynamic Low-Rank Adaptation (DoRA) method. DoRA decomposes high-rank LoRA
layers into structured single-rank components, allowing for dynamic pruning of
parameter budget based on their importance to specific tasks during training,
which makes the most of the limited parameter budget. Experimental results
demonstrate that DoRA can achieve competitive performance compared with LoRA
and full model fine-tuning, and outperform various strong baselines with the
same storage parameter budget. Our code is available at
this https URL

本研究介绍了动态低秩适应（DoRA）方法，该方法在训练过程中将高秩的低秩层分解为结构化的单秩元件，允许根据任务重要性动态修剪参数预算，以充分利用有限的参数预算，并且实验证明 DoRA 方法在与 LoRA 和完全模型微调相比具有竞争性的性能和优于各种强基线模型的结果。

DoRA: 通过动态排名分布增强参数效率微调

DoRA: Enhancing Parameter-Efficient Fine-Tuning with Dynamic Rank  Distribution

Finetuning large language models requires huge GPU memory, restricting the
choice to acquire Larger models. While the quantized version of the Low-Rank
Adaptation technique, named QLoRA, significantly alleviates this issue, finding
the efficient LoRA rank is still challenging. Moreover, QLoRA is trained on a
pre-defined rank and, therefore, cannot be reconfigured for its lower ranks
without requiring further fine-tuning steps. This paper proposes QDyLoRA
-Quantized Dynamic Low-Rank Adaptation-, as an efficient quantization approach
for dynamic low-rank adaptation. Motivated by Dynamic LoRA, QDyLoRA is able to
efficiently finetune LLMs on a set of pre-defined LoRA ranks. QDyLoRA enables
fine-tuning Falcon-40b for ranks 1 to 64 on a single 32 GB V100-GPU through one
round of fine-tuning. Experimental results show that QDyLoRA is competitive to
QLoRA and outperforms when employing its optimal rank.

QLoRA 和 QDyLoRA 是一种用于大型语言模型的有效量化方法，能够在 GPU 内存限制下进行适应性微调，并在寻找适合的低秩参数方面具有竞争力。

QDyLoRA：用于高效大型语言模型调优的量化动态低秩自适应

QDyLoRA: Quantized Dynamic Low-Rank Adaptation for Efficient Large  Language Model Tuning

With the ever-growing size of pre-trained models (PMs), fine-tuning them has
become more expensive and resource-hungry. As a remedy, low-rank adapters
(LoRA) keep the main pre-trained weights of the model frozen and just introduce
some learnable truncated SVD modules (so-called LoRA blocks) to the model.
While LoRA blocks are parameter efficient, they suffer from two major problems:
first, the size of these blocks is fixed and cannot be modified after training
(for example, if we need to change the rank of LoRA blocks, then we need to
re-train them from scratch); second, optimizing their rank requires an
exhaustive search and effort. In this work, we introduce a dynamic low-rank
adaptation (DyLoRA) technique to address these two problems together. Our
DyLoRA method trains LoRA blocks for a range of ranks instead of a single rank
by sorting out the representation learned by the adapter module at different
ranks during training. We evaluate our solution on different tasks of the GLUE
benchmark using the RoBERTa model. Our results show that we can train dynamic
search-free models with DyLoRA at least $7\times$ faster than LoRA without
significantly compromising performance. Moreover, our models can perform
consistently well on a much larger range of ranks compared to LoRA.

本文提出了一种名为 Dynamic Low-Rank Adaptation 的技术，旨在通过训练预训练模型的适配器模块，为多个秩数建立动态搜索免费的模型，并证明该方法可以显著加快训练速度并在 GLUE 基准测试中取得一致的优秀表现。