Large language models (LLMs) have demonstrated state-of-the-art performance
across various tasks. However, the latency of inference and the large GPU
memory consumption of LLMs restrict their deployment performance. Recently,
there have been some efficient attempts to quantize LLMs, yet inference with
large batch size or long sequence still has the issue of being compute-bound.
Fine-grained quantization methods have showcased their proficiency in achieving
low-bit quantization for LLMs, while requiring FP16 data type for linear layer
computations, which is time-consuming when dealing with large batch size or
long sequence. In this paper, we introduce a method called FlattenQuant, which
significantly reduces the maximum value of the tensor by flattening the large
channels in the tensor, to achieve low bit per-tensor quantization with minimal
accuracy loss. Our experiments show that FlattenQuant can directly use 4 bits
to achieve 48.29% of the linear layer calculation in LLMs, with the remaining
layers using 8 bits. The 4-bit matrix multiplication introduced in the
FlattenQuant method can effectively address the compute-bound caused by large
matrix calculation. Our work achieves up to 2$\times$ speedup and 2.3$\times$
memory reduction for LLMs with negligible loss in accuracy.

使用 FlattenQuant 方法在大型语言模型中实现低比特量化，通过将张量中的大通道展平以显著减少张量的最大值，本方法能够直接使用 4 位来实现 LLMs 中线性层计算的 48.29%，并使用 8 位来处理其余层，从而实现高达 2 倍的速度提升和 2.3 倍的内存减少，而准确度损失微不足道。