The high computational and memory requirements of large language model (LLM)
inference traditionally make it feasible only with multiple high-end
accelerators. Motivated by the emerging demand for latency-insensitive tasks
with batched processing, this paper initiates the study of high-throughput LLM
inference using limited resources, such as a single commodity GPU. We present
FlexGen, a high-throughput generation engine for running LLMs with limited GPU
memory. FlexGen can be flexibly configured under various hardware resource
constraints by aggregating memory and computation from the GPU, CPU, and disk.
Through a linear programming optimizer, it searches for efficient patterns to
store and access tensors. FlexGen further compresses these weights and the
attention cache to 4 bits with negligible accuracy loss. These techniques
enable FlexGen to have a larger space of batch size choices and thus
significantly increase maximum throughput. As a result, when running OPT-175B
on a single 16GB GPU, FlexGen achieves significantly higher throughput compared
to state-of-the-art offloading systems, reaching a generation throughput of 1
token/s for the first time with an effective batch size of 144. On the HELM
benchmark, FlexGen can benchmark a 30B model with a 16GB GPU on 7
representative sub-scenarios in 21 hours. The code is available at
this https URL

本文旨在研究如何利用有限的 GPU 资源进行高吞吐量的大语言模型推理，通过提出一种高吞吐量生成引擎 FlexGen，结合线性规划优化器并压缩权重和注意力缓存，成功实现在仅一个 16GB GPU 上运行 OPT-175B，并取得了较高的推理吞吐量。