Recently, considerable efforts have been directed towards compressing Large
Language Models (LLMs), which showcase groundbreaking capabilities across
diverse applications but entail significant deployment costs due to their large
sizes. Meanwhile, much less attention has been given to mitigating the costs
associated with deploying multiple LLMs of varying sizes despite its practical
significance. Thus, this paper introduces \emph{any-precision LLM}, extending
the concept of any-precision DNN to LLMs. Addressing challenges in
any-precision LLM, we propose a lightweight method for any-precision
quantization of LLMs, leveraging a post-training quantization framework, and
develop a specialized software engine for its efficient serving. As a result,
our solution significantly reduces the high costs of deploying multiple,
different-sized LLMs by overlaying LLMs quantized to varying bit-widths, such
as 3, 4, ..., $n$ bits, into a memory footprint comparable to a single $n$-bit
LLM. All the supported LLMs with varying bit-widths demonstrate
state-of-the-art model quality and inference throughput, proving itself to be a
compelling option for deployment of multiple, different-sized LLMs. The source
code will be publicly available soon.

该研究论文介绍了 Any-Precision LLM 的概念，并提出了一种轻量级的 Any-Precision 量化方法，以及对应的软件引擎，有效地解决了多个不同大小 LLM 的部署成本问题。