Quantization is an indispensable technique for serving Large Language Models
(LLMs) and has recently found its way into LoRA fine-tuning. In this work we
focus on the scenario where quantization and LoRA fine-tuning are applied
together on a pre-trained model. In such cases it is common to observe a
consistent gap in the performance on downstream tasks between full fine-tuning
and quantization plus LoRA fine-tuning approach. In response, we propose LoftQ
(LoRA-Fine-Tuning-aware Quantization), a novel quantization framework that
simultaneously quantizes an LLM and finds a proper low-rank initialization for
LoRA fine-tuning. Such an initialization alleviates the discrepancy between the
quantized and full-precision model and significantly improves the
generalization in downstream tasks. We evaluate our method on natural language
understanding, question answering, summarization, and natural language
generation tasks. Experiments show that our method is highly effective and
outperforms existing quantization methods, especially in the challenging 2-bit
and 2/4-bit mixed precision regimes. We will release our code.

我们提出了 LoftQ，这是一个新的量化框架，旨在同时量化 LLM 并为 LoRA fine-tuning 找到适当的低秩初始化，以改善量化和全精度模型之间的差异，并显著提高下游任务的泛化性能。

LoftQ: 针对大型语言模型的 LoRA 微调感知量化

LoftQ: LoRA-Fine-Tuning-Aware Quantization for Large Language Models

We propose a novel low-rank initialization framework for training low-rank
deep neural networks -- networks where the weight parameters are
re-parameterized by products of two low-rank matrices. The most successful
prior existing approach, spectral initialization, draws a sample from the
initialization distribution for the full-rank setting and then optimally
approximates the full-rank initialization parameters in the Frobenius norm with
a pair of low-rank initialization matrices via singular value decomposition.
Our method is inspired by the insight that approximating the function
corresponding to each layer is more important than approximating the parameter
values. We provably demonstrate that there is a significant gap between these
two approaches for ReLU networks, particularly as the desired rank of the
approximating weights decreases, or as the dimension of the inputs to the layer
increases (the latter point holds when the network width is super-linear in
dimension). Along the way, we provide the first provably efficient algorithm
for solving the ReLU low-rank approximation problem for fixed parameter rank
$r$ -- previously, it was unknown that the problem was computationally
tractable to solve even for rank $1$. We also provide a practical algorithm to
solve this problem which is no more expensive than the existing spectral
initialization approach, and validate our theory by training ResNet and
EfficientNet models (He et al., 2016; Tan & Le, 2019) on ImageNet (Russakovsky
et al., 2015).

我们提出了一种新的低秩初始化框架，用于训练低秩深度神经网络，证明了这种方法在 ReLU 网络中的有效性，并使用此方法训练了 ResNet 和 EfficientNet 模型。