Currently, there are many quantization methods appeared for LLM quantization,
yet few are user-friendly and easy to be deployed locally. Packages like
TensorRT and Quantohave many underlying structures and self-invoking internal
functions, which are not conducive to developers' personalized development and
learning for deployment. Therefore, we develop LLMEasyQuant, it is a package
aiming to for easy quantization deployment which is user-friendly and suitable
for beginners' learning.

LLMEasyQuant 是一个旨在简化量化部署的用户友好的包，适合初学者学习。

LLMEasyQuant - 一个易于使用的 LLM 量化工具包

LLMEasyQuant -- An Easy to Use Toolkit for LLM Quantization

Despite recent advances in LLM quantization, activation quantization remains
to be challenging due to the activation outliers. Conventional remedies, e.g.,
mixing precisions for different channels, introduce extra overhead and reduce
the speedup. In this work, we develop a simple yet effective strategy to
facilitate per-tensor activation quantization by preventing the generation of
problematic tokens. Precisely, we propose a method to find a set of key-value
cache, coined CushionCache, which mitigates outliers in subsequent tokens when
inserted as a prefix. CushionCache works in two steps: First, we greedily
search for a prompt token sequence that minimizes the maximum activation values
in subsequent tokens. Then, we further tune the token cache to regularize the
activations of subsequent tokens to be more quantization-friendly. The proposed
method successfully addresses activation outliers of LLMs, providing a
substantial performance boost for per-tensor activation quantization methods.
We thoroughly evaluate our method over a wide range of models and benchmarks
and find that it significantly surpasses the established baseline of per-tensor
W8A8 quantization and can be seamlessly integrated with the recent activation
quantization method.

提出了一种名为 CushionCache 的方法，通过防止生成问题标记，来促进每个张量的激活量化，成功解决了 LLMs 的激活离群值问题，并为每个张量的激活量化方法提供了显著的性能提升。