Post-training quantization (PTQ) reduces the memory footprint of LLMs by
quantizing weights to low-precision datatypes. Since LLM inference is usually
memory-bound, PTQ methods can improve inference throughput. Recent
state-of-the-art PTQ approaches have converged on using vector quantization
(VQ) to quantize multiple weights at once, which improves information
utilization through better shaping. However, VQ requires a codebook with size
exponential in the dimension. This limits current VQ-based PTQ works to low VQ
dimensions ($\le 8$) that in turn limit quantization quality. Here, we
introduce QTIP, which instead uses trellis coded quantization (TCQ) to achieve
ultra-high-dimensional quantization. TCQ uses a stateful decoder that separates
the codebook size from the bitrate and effective dimension. QTIP introduces a
spectrum of lookup-only to computed lookup-free trellis codes designed for a
hardware-efficient "bitshift" trellis structure; these codes achieve
state-of-the-art results in both quantization quality and inference speed.

QTIP 使用三叶栅编码量化与硬件高效的 “比特移位” 三叶栅结构，实现了具有最先进的量化质量和推理速度的超高维度量化。