This article optimizes the inference performance of the Qwen-1.8B model by
performing Int8 quantization, vectorizing some operators in llama.cpp, and
modifying the compilation script to improve the compiler optimization level. On
the Yitian 710 experimental platform, the prefill performance is increased by
1.6 times, the decoding performance is increased by 24 times, the memory usage
is reduced to 1/5 of the original, and the accuracy loss is almost negligible.

通过进行 Int8 量化，对 llama.cpp 中的一些运算符进行矢量化，并修改编译脚本以提高编译器优化水平，优化了 Qwen-1.8B 模型的推断性能。在 Yitian 710 实验平台上，填充性能提高了 1.6 倍，解码性能提高了 24 倍，内存使用量减少到原来的 1/5，准确率损失几乎可以忽略不计。

基于 Llama.cpp 的 Armv9 架构通用大语言模型推理性能优化

Optimization of Armv9 architecture general large language model  inference performance based on Llama.cpp

Large language models (LLMs) show excellent performance but are compute- and
memory-intensive. Quantization can reduce memory and accelerate inference.
However, for LLMs beyond 100 billion parameters, existing methods cannot
maintain accuracy or do not run efficiently on hardware. We propose
SmoothQuant, a training-free, accuracy-preserving, and general-purpose
post-training quantization (PTQ) solution to enable 8-bit weight, 8-bit
activation (W8A8) quantization for LLMs. Based on the fact that weights are
easy to quantize while activations are not, SmoothQuant smooths the activation
outliers by offline migrating the quantization difficulty from activations to
weights with a mathematically equivalent transformation. SmoothQuant enables an
INT8 quantization of both weights and activations for all the matrix
multiplications in LLMs, including OPT-175B, BLOOM-176B, GLM-130B, and MT-NLG
530B. SmoothQuant has better hardware efficiency than existing techniques. We
demonstrate up to 1.56x speedup and 2x memory reduction for LLMs with
negligible loss in accuracy. We integrate SmoothQuant into FasterTransformer, a
state-of-the-art LLM serving framework, and achieve faster inference speed with
half the number of GPUs compared to FP16, enabling the serving of a 530B LLM
within a single node. Our work offers a turn-key solution that reduces hardware
costs and democratizes LLMs. Code is available at
this https URL.

SmoothQuant 是一个训练免费的精度保持、通用的后训练量化解决方案，用于大型语言模型（LLMs），通过平滑激活异常值并在权重和激活之间进行数学上等效的变换以迁移量化难度，可以实现 LLMs 的 8 位权重和激活（W8A8）量化，同时提高硬件效率，以较小的精度损失实现高达 2 倍的内存减少和 1.56 倍的加速，是一个可降低硬件成本、民主化 LLMs 的一站式解决方案。

SmoothQuant：大型语言模型后训练量化的准确高效算法

SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models

Researches have demonstrated that low bit-width (e.g., INT8) quantization can
be employed to accelerate the inference process. It makes the gradient
quantization very promising since the backward propagation requires
approximately twice more computation than forward one. Due to the variability
and uncertainty of gradient distribution, a lot of methods have been proposed
to attain training stability. However, most of them ignore the channel-wise
gradient distributions and the impact of gradients with different magnitudes,
resulting in the degradation of final accuracy. In this paper, we propose a
novel INT8 quantization training framework for convolutional neural network to
address the above issues. Specifically, we adopt Gradient Vectorized
Quantization to quantize the gradient, based on the observation that layer-wise
gradients contain multiple distributions along the channel dimension. Then,
Magnitude-aware Clipping Strategy is introduced by taking the magnitudes of
gradients into consideration when minimizing the quantization error, and we
present a theoretical derivation to solve the quantization parameters of
different distributions. Experimental results on broad range of computer vision
tasks, such as image classification, object detection and video classification,
demonstrate that the proposed Distribution Adaptive INT8 Quantization training
method has achieved almost lossless training accuracy for different backbones,
including ResNet, MobileNetV2, InceptionV3, VGG and AlexNet, which is superior
to the state-of-the-art techniques. Moreover, we further implement the INT8
kernel that can accelerate the training iteration more than 200% under the
latest Turing architecture, i.e., our method excels on both training accuracy
and speed.

本文提出了一种新的 INT8 量化训练框架，其中采用梯度矢量化量化梯度，并引入基于梯度幅度的裁剪策略来解决量化梯度的误差，实验结果表明，在图像分类、物体检测和视频分类等广泛的计算机视觉任务中，所提出的 Distribution Adaptive INT8 Quantization 训练方法已经达到了几乎无损的训练精度，优于现有技术，并且这种方法能够在最新的 Turing 架构下实现训练速度提高 200% 以上。