We introduce Integer Scale, a novel post-training quantization scheme for large language models that effectively resolves the inference bottleneck in current fine-grained quantization approaches while maintaining similar accuracies. Integer Scale is a free lunch as it requires no extra calibration or fine-tuning which will otherwise incur additional costs. It can be used plug-and-play for most fine-grained quantization methods. Its integration results in at most 1.85x end-to-end speed boost over the original counterpart with comparable accuracy. Additionally, due to the orchestration of the proposed Integer Scale and fine-grained quantization, we resolved the quantization difficulty for Mixtral-8x7B and LLaMA-3 models with negligible performance degradation, and it comes with an end-to-end speed boost of 2.13x, and 2.31x compared with their FP16 versions respectively.

我们介绍了一种新的后训练量化方案——整数比例，它有效地解决了当前细粒度量化方法中的推理瓶颈问题，同时保持了类似的准确性。整数比例不需要额外的校准或微调，从而避免了额外成本。它可以与大多数细粒度量化方法直接使用，最多可使原始模型的端到端速度提升1.85倍，并且能够解决Mixtral-8x7B和LLaMA-3模型的量化困难，几乎不降低性能，并分别相对于它们的FP16版本提供2.13倍和2.31倍的端到端速度提升。

整数量化尺度：加速低位宽量化LLM的免费午餐