We introduce Integer Scale, a novel post-training quantization scheme for
large language models that effectively resolves the inference bottleneck in
current fine-grained quantization approaches while maintaining similar
accuracies. Integer Scale is a free lunch as it requires no extra calibration
or fine-tuning which will otherwise incur additional costs. It can be used
plug-and-play for most fine-grained quantization methods. Its integration
results in at most 1.85x end-to-end speed boost over the original counterpart
with comparable accuracy. Additionally, due to the orchestration of the
proposed Integer Scale and fine-grained quantization, we resolved the
quantization difficulty for Mixtral-8x7B and LLaMA-3 models with negligible
performance degradation, and it comes with an end-to-end speed boost of 2.13x,
and 2.31x compared with their FP16 versions respectively.

我们介绍了一种新的后训练量化方案 —— 整数比例，它有效地解决了当前细粒度量化方法中的推理瓶颈问题，同时保持了类似的准确性。整数比例不需要额外的校准或微调，从而避免了额外成本。它可以与大多数细粒度量化方法直接使用，最多可使原始模型的端到端速度提升 1.85 倍，并且能够解决 Mixtral-8x7B 和 LLaMA-3 模型的量化困难，几乎不降低性能，并分别相对于它们的 FP16 版本提供 2.13 倍和 2.31 倍的端到端速度提升。