The large language model era urges faster and less costly inference. Prior
model compression works on LLMs tend to undertake a software-centric approach
primarily focused on the simulated quantization performance. By neglecting the
feasibility of deployment, these approaches are typically disabled in real
practice. They used to drastically push down the quantization bit range for a
reduced computation which might not be supported by the mainstream hardware, or
involve sophisticated algorithms that introduce extra computation or memory
access overhead. We argue that pursuing a hardware-centric approach in the
construction of quantization algorithms is crucial. In this regard, we are
driven to build our compression method on top of hardware awareness,
eliminating impractical algorithm choices while maximizing the benefit of
hardware acceleration. Our method, OdysseyLLM, comes with a novel W4A8 kernel
implementation called FastGEMM and a combined recipe of quantization
strategies. Extensive experiments manifest the superiority of our W4A8 method
which brings the actual speed boosting up to \textbf{4$\times$} compared to
Hugging Face FP16 inference and \textbf{2.23$\times$} vs. the state-of-the-art
inference engine TensorRT-LLM in FP16, and \textbf{1.45$\times$} vs.
TensorRT-LLM in INT8, yet without substantially harming the performance.

通过硬件为中心的方法，我们的压缩方法在硬件加速的基础上构建了一种新的 W4A8 内核实现，具有量化策略的综合配方，通过广泛的实验证明了我们的 W4A8 方法对于 Hugging Face FP16 推断的实际加速效果为 4 倍，对于 TensorRT-LLM 推断引擎的 FP16 加速效果为 2.23 倍，对于 TensorRT-LLM 推断引擎的 INT8 加速效果为 1.45 倍，且不会对性能造成实质性的损害。

可部署的 LLM 压缩加速之旅

A Speed Odyssey for Deployable Quantization of LLMs

Federated Learning (FL) is a powerful technique for training a model on a
server with data from several clients in a privacy-preserving manner. In FL, a
server sends the model to every client, who then train the model locally and
send it back to the server. The server aggregates the updated models and
repeats the process for several rounds. FL incurs significant communication
costs, in particular when transmitting the updated local models from the
clients back to the server. Recently proposed algorithms quantize the model
parameters to efficiently compress FL communication. These algorithms typically
have a quantization level that controls the compression factor. We find that
dynamic adaptations of the quantization level can boost compression without
sacrificing model quality. First, we introduce a time-adaptive quantization
algorithm that increases the quantization level as training progresses. Second,
we introduce a client-adaptive quantization algorithm that assigns each
individual client the optimal quantization level at every round. Finally, we
combine both algorithms into DAdaQuant, the doubly-adaptive quantization
algorithm. Our experiments show that DAdaQuant consistently improves
client$\rightarrow$server compression, outperforming the strongest non-adaptive
baselines by up to $2.8\times$.

本研究介绍了一种动态自适应的量化算法 DAdaQuant，可以在保证模型质量的前提下提高 client 到 server 的压缩比例，比强非自适应基线算法提高多达 2.8 倍。

DAdaQuant：用于通信高效联邦学习的双重自适应量化

DAdaQuant: Doubly-adaptive quantization for communication-efficient  Federated Learning

We consider the problem of distributed feature quantization, where the goal
is to enable a pretrained classifier at a central node to carry out its
classification on features that are gathered from distributed nodes through
communication constrained channels. We propose the design of distributed
quantization schemes specifically tailored to the classification task: unlike
quantization schemes that help the central node reconstruct the original signal
as accurately as possible, our focus is not reconstruction accuracy, but
instead correct classification. Our work does not make any apriori
distributional assumptions on the data, but instead uses training data for the
quantizer design. Our main contributions include: we prove NP-hardness of
finding optimal quantizers in the general case; we design an optimal scheme for
a special case; we propose quantization algorithms, that leverage discrete
neural representations and training data, and can be designed in
polynomial-time for any number of features, any number of classes, and
arbitrary division of features across the distributed nodes. We find that
tailoring the quantizers to the classification task can offer significant
savings: as compared to alternatives, we can achieve more than a factor of two
reduction in terms of the number of bits communicated, for the same
classification accuracy.

本文提出了一种分布式量化方案，旨在用训练数据为分类器设计量化器，不需要先验数据假设，而是专注于正确分类，可在任何特征数、任何类别数和任意特征分布的情况下进行多项式时间量化算法的设计。结果表明，将量化器适应于分类任务可以获得显着的节省，并可在保持预测精度不变的情况下将比特通信数量减少超过两倍。