We introduce Integer Scale, a novel post-training quantization scheme for
large language models that effectively resolves the inference bottleneck in
current fine-grained quantization approaches while maintaining similar
accuracies. Integer Scale is a free lunch as it requires no extra calibration
or fine-tuning which will otherwise incur additional costs. It can be used
plug-and-play for most fine-grained quantization methods. Its integration
results in at most 1.85x end-to-end speed boost over the original counterpart
with comparable accuracy. Additionally, due to the orchestration of the
proposed Integer Scale and fine-grained quantization, we resolved the
quantization difficulty for Mixtral-8x7B and LLaMA-3 models with negligible
performance degradation, and it comes with an end-to-end speed boost of 2.13x,
and 2.31x compared with their FP16 versions respectively.

我们介绍了一种新的后训练量化方案 —— 整数比例，它有效地解决了当前细粒度量化方法中的推理瓶颈问题，同时保持了类似的准确性。整数比例不需要额外的校准或微调，从而避免了额外成本。它可以与大多数细粒度量化方法直接使用，最多可使原始模型的端到端速度提升 1.85 倍，并且能够解决 Mixtral-8x7B 和 LLaMA-3 模型的量化困难，几乎不降低性能，并分别相对于它们的 FP16 版本提供 2.13 倍和 2.31 倍的端到端速度提升。

整数量化尺度：加速低位宽量化 LLM 的免费午餐

Integer Scale: A Free Lunch for Faster Fine-grained Quantization of LLMs

We propose a novel fine-grained quantization (FGQ) method to ternarize
pre-trained full precision models, while also constraining activations to 8 and
4-bits. Using this method, we demonstrate a minimal loss in classification
accuracy on state-of-the-art topologies without additional training. We provide
an improved theoretical formulation that forms the basis for a higher quality
solution using FGQ. Our method involves ternarizing the original weight tensor
in groups of $N$ weights. Using $N=4$, we achieve Top-1 accuracy within $3.7\%$
and $4.2\%$ of the baseline full precision result for Resnet-101 and Resnet-50
respectively, while eliminating $75\%$ of all multiplications. These results
enable a full 8/4-bit inference pipeline, with best-reported accuracy using
ternary weights on ImageNet dataset, with a potential of $9\times$ improvement
in performance. Also, for smaller networks like AlexNet, FGQ achieves
state-of-the-art results. We further study the impact of group size on both
performance and accuracy. With a group size of $N=64$, we eliminate
$\approx99\%$ of the multiplications; however, this introduces a noticeable
drop in accuracy, which necessitates fine tuning the parameters at lower
precision. We address this by fine-tuning Resnet-50 with 8-bit activations and
ternary weights at $N=64$, improving the Top-1 accuracy to within $4\%$ of the
full precision result with $<30\%$ additional training overhead. Our final
quantized model can run on a full 8-bit compute pipeline using 2-bit weights
and has the potential of up to $15\times$ improvement in performance compared
to baseline full-precision models.

本文提出了一种精细的量化方法 (Fine-grained Quantization, FGQ)，该方法可对预训练的全精度模型进行三值化，同时将激活限制为 8 位和 4 位。通过该方法，我们证明了无需额外训练，就可以在最先进的拓扑结构上实现最小分类精度损失。该方法可用于 Resnet-101 和 Resnet-50 等模型，可以消除 75% 的乘法运算，从而实现完整的 8/4 位推理管道，并在 ImageNet 数据集上实现最佳报告精度，性能提升潜力高达 9 倍。最终的量化模型可以在全精度模型的基础上提高 15 倍的性能。