The recent advance of self-supervised learning associated with the Transformer architecture enables natural language processing (NLP) to exhibit extremely low perplexity. Such powerful models demand ever-increasing model size, and thus, large amounts of computations and memory footprints. In this paper, we propose an efficient inference framework for large-scale generative language models. As the key to reducing model size, we quantize weights by a non-uniform quantization method. Then, quantized matrix multiplications are accelerated by our proposed kernel, called nuQmm, which allows a wide trade-off between compression ratio and accuracy. Our proposed nuQmm reduces the latency of not only each GPU but also the entire inference of large LMs because a high compression ratio (by low-bit quantization) mitigates the minimum required number of GPUs. We demonstrate that nuQmm can accelerate the inference speed of the GPT-3 (175B) model by about 14.4 times and save energy consumption by 93%.

该论文提出了一个有效的推理框架用于大规模生成式语言模型，采用自我监督学习和Transformer架构的最新进展实现了低困惑度，通过非均匀量化和nuQmm的加速矩阵乘法，实现了模型大小的减小，并减少了大型LM的推理延迟

nuQmm: 大规模生成式语言模型高效推理的量化矩阵乘法