Recent breakthroughs in Large-scale language models (LLMs) have demonstrated impressive performance on various tasks. The immense sizes of LLMs have led to very high resource demand and cost for running the models. Though the models are largely served using uniform high-caliber GPUs nowadays, utilizing a heterogeneous cluster with a mix of available high- and low-capacity GPUs can potentially substantially reduce the serving cost. There is a lack of designs to support efficient LLM serving using a heterogeneous cluster, while the current solutions focus on model partition and uniform compression among homogeneous devices. This paper proposes LLM-PQ, a system that advocates adaptive model quantization and phase-aware partition to improve LLM serving efficiency on heterogeneous GPU clusters. We carefully decide on mixed-precision model quantization together with phase-aware model partition and micro-batch sizing in distributed LLM serving with an efficient algorithm, to greatly enhance inference throughput while fulfilling user-specified model quality targets. Extensive experiments on production inference workloads in 11 different clusters demonstrate that LLM-PQ achieves up to 2.88x (2.26x on average) throughput improvement in inference, showing great advantages over state-of-the-art works.

LLM-PQ是一个提倡自适应模型量化和阶段感知模型分区的系统，旨在通过在异构GPU集群上提高LLM的服务效率。通过混合精度模型量化、阶段感知模型分区和微批量大小的决策，LLM-PQ大大提高了推理吞吐量，同时满足用户指定的模型质量目标。对11个不同集群上的生产推理工作负载进行的大量实验表明，LLM-PQ在推理上实现了高达2.88倍（平均2.26倍）的吞吐量改进，显示出较其他最先进方法的巨大优势。

LLM-PQ：利用多阶段感知分割和自适应量化为异构集群提供LLM