Large Language Models (LLMs) have greatly advanced the natural language
processing paradigm. However, the high computational load and huge model sizes
pose a grand challenge for deployment on edge devices. To this end, we propose
APTQ (Attention-aware Post-Training Mixed-Precision Quantization) for LLMs,
which considers not only the second-order information of each layer's weights,
but also, for the first time, the nonlinear effect of attention outputs on the
entire model. We leverage the Hessian trace as a sensitivity metric for
mixed-precision quantization, ensuring an informed precision reduction that
retains model performance. Experiments show APTQ surpasses previous
quantization methods, achieving an average of 4 bit width a 5.22 perplexity
nearly equivalent to full precision in the C4 dataset. In addition, APTQ
attains state-of-the-art zero-shot accuracy of 68.24\% and 70.48\% at an
average bitwidth of 3.8 in LLaMa-7B and LLaMa-13B, respectively, demonstrating
its effectiveness to produce high-quality quantized LLMs.

通过引入 APTQ（关注感知的后训练混合精度量化），该研究提出了一种在大规模语言模型上进行混合精度量化的方法，利用 Hessian 迹作为灵敏度指标，以实现在模型性能保持的前提下进行精度降低，并取得了优于以往量化方法的效果。

APTQ：针对大型语言模型的注意力感知后训练混合精度量化

APTQ: Attention-aware Post-Training Mixed-Precision Quantization for  Large Language Models

We consider finding flat, local minimizers by adding average weight
perturbations. Given a nonconvex function $f: \mathbb{R}^d \rightarrow
\mathbb{R}$ and a $d$-dimensional distribution $\mathcal{P}$ which is symmetric
at zero, we perturb the weight of $f$ and define $F(W) = \mathbb{E}[f({W +
U})]$, where $U$ is a random sample from $\mathcal{P}$. This injection induces
regularization through the Hessian trace of $f$ for small, isotropic Gaussian
perturbations. Thus, the weight-perturbed function biases to minimizers with
low Hessian trace. Several prior works have studied settings related to this
weight-perturbed function by designing algorithms to improve generalization.
Still, convergence rates are not known for finding minima under the average
perturbations of the function $F$. This paper considers an SGD-like algorithm
that injects random noise before computing gradients while leveraging the
symmetry of $\mathcal{P}$ to reduce variance. We then provide a rigorous
analysis, showing matching upper and lower bounds of our algorithm for finding
an approximate first-order stationary point of $F$ when the gradient of $f$ is
Lipschitz-continuous. We empirically validate our algorithm for several image
classification tasks with various architectures. Compared to sharpness-aware
minimization, we note a 12.6% and 7.8% drop in the Hessian trace and top
eigenvalue of the found minima, respectively, averaged over eight datasets.
Ablation studies validate the benefit of the design of our algorithm.

通过注入随机噪声并利用对称分布来降低方差，提出一种类 SGD 算法来求解期望函数的近似一阶稳定点，用于图像分类任务中，相较于尖锐性感知最小化，求得的解的 Hessian 跟踪和特征值较低。

最优收敛速率下的平坦极小值噪音稳定优化

Noise Stability Optimization for Flat Minima with Optimal Convergence  Rates

Neural network pruning is a practical way for reducing the size of trained
models and the number of floating-point operations. One way of pruning is to
use the relative Hessian trace to calculate sensitivity of each channel, as
compared to the more common magnitude pruning approach. However, the stochastic
approach used to estimate the Hessian trace needs to iterate over many times
before it can converge. This can be time-consuming when used for larger models
with many millions of parameters. To address this problem, we modify the
existing approach by estimating the Hessian trace using FP16 precision instead
of FP32. We test the modified approach (EHAP) on
ResNet-32/ResNet-56/WideResNet-28-8 trained on CIFAR10/CIFAR100 image
classification tasks and achieve faster computation of the Hessian trace.
Specifically, our modified approach can achieve speed ups ranging from 17% to
as much as 44% during our experiments on different combinations of model
architectures and GPU devices. Our modified approach also takes up around 40%
less GPU memory when pruning ResNet-32 and ResNet-56 models, which allows for a
larger Hessian batch size to be used for estimating the Hessian trace.
Meanwhile, we also present the results of pruning using both FP16 and FP32
Hessian trace calculation and show that there are no noticeable accuracy
differences between the two. Overall, it is a simple and effective way to
compute the relative Hessian trace faster without sacrificing on pruned model
performance. We also present a full pipeline using EHAP and quantization aware
training (QAT), using INT8 QAT to compress the network further after pruning.
In particular, we use symmetric quantization for the weights and asymmetric
quantization for the activations.

使用 FP16 精度代替 FP32 实现变异矩阵迹快速估算的剪枝方法可以加快计算速度并缩小 GPU 内存占用，在不影响模型性能的情况下，可以进一步使用 QAT 技术对模型进行压缩。

基于 Hessian 剪枝的资源高效神经网络

Resource Efficient Neural Networks Using Hessian Based Pruning

Policy gradient algorithms are an important family of deep reinforcement
learning techniques. Many past research endeavors focused on using the
first-order policy gradient information to train policy networks. Different
from these works, we conduct research in this paper driven by the believe that
properly utilizing and controlling Hessian information associated with the
policy gradient can noticeably improve the performance of policy gradient
algorithms. One key Hessian information that attracted our attention is the
Hessian trace, which gives the divergence of the policy gradient vector field
in the Euclidean policy parametric space. We set the goal to generalize this
Euclidean policy parametric space into a general Riemmanian manifold by
introducing a metric tensor field $g_ab$ in the parametric space. This is
achieved through newly developed mathematical tools, deep learning algorithms,
and metric tensor deep neural networks (DNNs). Armed with these technical
developments, we propose a new policy gradient algorithm that learns to
minimize the absolute divergence in the Riemannian manifold as an important
regularization mechanism, allowing the Riemannian manifold to smoothen its
policy gradient vector field. The newly developed algorithm is experimentally
studied on several benchmark reinforcement learning problems. Our experiments
clearly show that the new metric tensor regularized algorithm can significantly
outperform its counterpart that does not use our regularization technique.
Additional experimental analysis further suggests that the trained metric
tensor DNN and the corresponding metric tensor $g_{ab}$ can effectively reduce
the absolute divergence towards zero in the Riemannian manifold.

本文提出了一种新的策略梯度算法，使用矩阵张量分解以及 Riemann 流形，通过最小化 Riemann 流形中的绝对发散来规范化算法，实现了深度强化学习算法性能的显著提升。