We introduce QUICK, a group of novel optimized CUDA kernels for the efficient
inference of quantized Large Language Models (LLMs). QUICK addresses the shared
memory bank-conflict problem of state-of-the-art mixed precision matrix
multiplication kernels. Our method interleaves the quantized weight matrices of
LLMs offline to skip the shared memory write-back after the dequantization. We
demonstrate up to 1.91x speedup over existing kernels of AutoAWQ on larger
batches and up to 1.94x throughput gain on representative LLM models on various
NVIDIA GPU devices.

QUICK 是一组新型优化的 CUDA 内核，用于高效推理量化的大型语言模型。QUICK 解决了现有混合精度矩阵乘法内核中的共享内存冲突问题，通过离线交错量化的权重矩阵跳过解量化后的共享内存写回。我们展示了在较大批次上相对于 AutoAWQ 现有内核的高达 1.91 倍的加速效果，并在各种 NVIDIA GPU 设备上相对于代表性 LLM 模型获得高达 1.94 倍的吞吐量提升。