Recently, the number of parameters in DNNs has explosively increased, as
exemplified by LLMs (Large Language Models), making inference on small-scale
computers more difficult. Model compression technology is, therefore, essential
for integration into products. In this paper, we propose a method of
quantization-aware training. We introduce a novel normalization (Layer-Batch
Normalization) that is independent of the mini-batch size and does not require
any additional computation cost during inference. Then, we quantize the weights
by the scaled round-clip function with the weight standardization. We also
quantize activation functions using the same function and apply surrogate
gradients to train the model with both quantized weights and the quantized
activation functions. We call this method Magic for the age of Quantised DNNs
(MaQD). Experimental results show that our quantization method can be achieved
with minimal accuracy degradation.

本文提出了一种量化感知训练的方法，通过引入一种独立于小批量大小的新型规范化（Layer-Batch Normalization）和标准化权重的缩放环夹函数对权重进行量化，同时对激活函数使用同样的函数进行量化，并应用替代梯度来训练模型，实验证明我们的量化方法可以在最小的准确性降低下实现。

量化 DNN 时代的魔法

Magic for the Age of Quantized DNNs

We propose BinaryRelax, a simple two-phase algorithm, for training deep
neural networks with quantized weights. The set constraint that characterizes
the quantization of weights is not imposed until the late stage of training,
and a sequence of \emph{pseudo} quantized weights is maintained. Specifically,
we relax the hard constraint into a continuous regularizer via Moreau envelope,
which turns out to be the squared Euclidean distance to the set of quantized
weights. The pseudo quantized weights are obtained by linearly interpolating
between the float weights and their quantizations. A continuation strategy is
adopted to push the weights towards the quantized state by gradually increasing
the regularization parameter. In the second phase, exact quantization scheme
with a small learning rate is invoked to guarantee fully quantized weights. We
test BinaryRelax on the benchmark CIFAR and ImageNet color image datasets to
demonstrate the superiority of the relaxed quantization approach and the
improved accuracy over the state-of-the-art training methods. Finally, we prove
the convergence of BinaryRelax under an approximate orthogonality condition.

该论文提出了一种名为 BinaryRelax 的算法，用于训练具有量化权重的深度神经网络，该方法使用 Moreau 包络将硬约束放宽为连续正则化器，并通过逐步调整正则化参数将权重推向量化状态，经过在 CIFAR 和 ImageNet 基准数据集上的实验，表明该算法在训练效果方面具有优越性。