Fine-tuning is a crucial process for adapting large language models (LLMs) to
diverse applications. In certain scenarios, such as multi-tenant serving,
deploying multiple LLMs becomes necessary to meet complex demands. Recent
studies suggest decomposing a fine-tuned LLM into a base model and
corresponding delta weights, which are then compressed using low-rank or
low-bit approaches to reduce costs. In this work, we observe that existing
low-rank and low-bit compression methods can significantly harm the model
performance for task-specific fine-tuned LLMs (e.g., WizardMath for math
problems). Motivated by the long-tail distribution of singular values in the
delta weights, we propose a delta quantization approach using mixed-precision.
This method employs higher-bit representation for singular vectors
corresponding to larger singular values. We evaluate our approach on various
fine-tuned LLMs, including math LLMs, code LLMs, chat LLMs, and even VLMs.
Experimental results demonstrate that our approach performs comparably to full
fine-tuned LLMs, surpassing both low-rank and low-bit baselines by a
considerable margin. Additionally, we show that our method is compatible with
various backbone LLMs, such as Llama-2, Llama-3, and Mistral, highlighting its
generalizability.

通过将细调的大语言模型进行分解和压缩，并采用混合精度的方法，我们提出了一个保持模型性能的增量量化方法，实验证明其在各种细调的语言模型中表现出与完整模型相当的性能，在低秩和低比特方法上也表现出明显优势，并且与其他基础语言模型兼容。