The inference of Large language models (LLMs) requires immense computation
and memory resources. To curtail these costs, quantisation has merged as a
promising solution, but existing LLM quantisation mainly focuses on 8-bit. In
this work, we explore the statistical and learning properties of the LLM layer
and attribute the bottleneck of LLM quantisation to numerical scaling offsets.
To address this, we adapt block quantisations for LLMs, a family of methods
that share scaling factors across packed numbers. Block quantisations
efficiently reduce the numerical scaling offsets solely from an arithmetic
perspective, without additional treatments in the computational path. Our
nearly-lossless quantised 6-bit LLMs achieve a $19\times$ higher arithmetic
density and $5\times$ memory density than the float32 baseline, surpassing the
prior art 8-bit quantisation by $2.5\times$ in arithmetic density and
$1.2\times$ in memory density, without requiring any data calibration or
re-training. We also share our insights into sub-8-bit LLM quantisation,
including the mismatch between activation and weight distributions, optimal
fine-tuning strategies, and a lower quantisation granularity inherent in the
statistical properties of LLMs. The latter two tricks enable nearly-lossless
4-bit LLMs on downstream tasks. The proposed framework will be open-sourced
upon publication.

通过使用块量化技术，本研究实现了基于大型语言模型（LLMs）的几乎无损 6 位量化，与浮点数基准相比算术密度增加了 19 倍，内存密度增加了 5 倍，超过了之前 8 位量化的先前技术。同时，介绍了子 8 位 LLM 量化的关键见解，包括激活和权重分布之间的不匹配、最佳的微调策略，以及 LLMs 统计属性中固有的较低量化粒度。该研究提出的框架将在发表后开源。