The prohibitive sizes of Large Language Models (LLMs) today make it difficult
to deploy them on memory-constrained edge devices. This work introduces $\rm
CALDERA$ -- a new post-training LLM compression algorithm that harnesses the
inherent low-rank structure of a weight matrix $\mathbf{W}$ by approximating it
via a low-rank, low-precision decomposition as $\mathbf{W} \approx \mathbf{Q} +
\mathbf{L}\mathbf{R}$. Here, $\mathbf{L}$ and $\mathbf{R}$ are low rank
factors, and the entries of $\mathbf{Q}$, $\mathbf{L}$ and $\mathbf{R}$ are
quantized. The model is compressed by substituting each layer with its
$\mathbf{Q} + \mathbf{L}\mathbf{R}$ decomposition, and the zero-shot
performance of the compressed model is evaluated. Additionally, $\mathbf{L}$
and $\mathbf{R}$ are readily amenable to low-rank adaptation, consequently
enhancing the zero-shot performance. $\rm CALDERA$ obtains this decomposition
by formulating it as an optimization problem
$\min_{\mathbf{Q},\mathbf{L},\mathbf{R}}\lVert(\mathbf{Q} +
\mathbf{L}\mathbf{R} - \mathbf{W})\mathbf{X}^\top\rVert_{\rm F}^2$, where
$\mathbf{X}$ is the calibration data, and $\mathbf{Q}, \mathbf{L}, \mathbf{R}$
are constrained to be representable using low-precision formats. Theoretical
upper bounds on the approximation error of $\rm CALDERA$ are established using
a rank-constrained regression framework, and the tradeoff between compression
ratio and model performance is studied by analyzing the impact of target rank
and quantization bit budget. Results illustrate that compressing LlaMa-$2$
$7$B/$70$B and LlaMa-$3$ $8$B models obtained using $\rm CALDERA$ outperforms
existing post-training LLM compression techniques in the regime of less than
$2.5$ bits per parameter. The implementation is available at:
\href{https://github.com/pilancilab/caldera}{this https URL}.

CALDERA 是一种新的后训练 LLM 压缩算法，通过近似低秩、低精度分解的权重矩阵近似表达模型，研究了压缩比和模型性能的权衡以及在小于 2.5 比特每参数的情况下相比现有的压缩技术的优越性。

使用低秩和低精度分解压缩大型语言模型

Compressing Large Language Models using Low Rank and Low Precision  Decomposition

Given the current trend of increasing size and complexity of machine learning
architectures, it has become of critical importance to identify new approaches
to improve the computational efficiency of model training. In this context, we
address the advantages of floating-point over fixed-point representation, and
present an in-depth study on the use of 8-bit floating-point number formats for
activations, weights, and gradients for both training and inference. We explore
the effect of different bit-widths for exponents and significands and different
exponent biases. The experimental results demonstrate that a suitable choice of
these low-precision formats enables faster training and reduced power
consumption without any degradation in accuracy for a range of deep learning
models for image classification and language processing.

该研究探讨了使用低精度浮点数表示法（例如 8 位浮点数格式）对深度学习模型进行训练和推理的影响，结果表明合适的低精度格式选择可加快训练和降低功耗，而不会降低准确性。