Efficient deployment of large language models (LLMs) necessitates low-bit
quantization to minimize model size and inference cost. While low-bit integer
formats (e.g., INT8/INT4) have been the conventional choice, emerging low-bit
floating-point formats (e.g., FP8/FP4) offer a compelling alternative and are
gaining support from cutting-edge hardware, such as NVIDIA's H100 GPU. However,
the superiority of low-bit INT versus FP formats for quantization on LLMs
remains unclear. In this study, we conduct a comparative analysis of INT and FP
quantization with the same bit-width, revealing that the optimal quantization
format varies across different layers due to the complexity and diversity of
tensor distribution. Consequently, we advocate the Mixture of Formats
Quantization (MoFQ), which selects the optimal format on a layer-wise basis.
This simple yet effective approach achieves state-of-the-art results in both
weight-only (W-only) and weight-activation (WA) post-training quantization
scenarios when tested on LLaMA across various tasks. In 4-bit W-only
quantization, MoFQ surpasses GPTQ without complex hyperparameter tuning and
with an order of magnitude faster quantization speed. While in 8-bit WA
quantization, MoFQ significantly outperforms INT/FP-only methods, achieving
performance close to the full precision model. Notably, MoFQ incurs no hardware
overhead compared to INT/FP-only quantization, as the bit-width remains
unchanged.

本研究比较研究了 INT 和 FP 低位量化在 LLMs 中的优劣，发现由于张量分布的复杂性和异质性，最优量化格式因层而异，提出了 MoFQ，这种简单易行的方法在各种任务中取得了最新的最佳结果，并且在不引入硬件开销的情况下具有显著的性能改进。