We address the problem of network quantization, that is, reducing bit-widths
of weights and/or activations to lighten network architectures. Quantization
methods use a rounding function to map full-precision values to the nearest
quantized ones, but this operation is not differentiable. There are mainly two
approaches to training quantized networks with gradient-based optimizers.
First, a straight-through estimator (STE) replaces the zero derivative of the
rounding with that of an identity function, which causes a gradient mismatch
problem. Second, soft quantizers approximate the rounding with continuous
functions at training time, and exploit the rounding for quantization at test
time. This alleviates the gradient mismatch, but causes a quantizer gap
problem. We alleviate both problems in a unified framework. To this end, we
introduce a novel quantizer, dubbed a distance-aware quantizer (DAQ), that
mainly consists of a distance-aware soft rounding (DASR) and a temperature
controller. To alleviate the gradient mismatch problem, DASR approximates the
discrete rounding with the kernel soft argmax, which is based on our insight
that the quantization can be formulated as a distance-based assignment problem
between full-precision values and quantized ones. The controller adjusts the
temperature parameter in DASR adaptively according to the input, addressing the
quantizer gap problem. Experimental results on standard benchmarks show that
DAQ outperforms the state of the art significantly for various bit-widths
without bells and whistles.

该论文介绍了一种统一的框架来解决网络量化问题，通过引入一种新型的距离感知量化器 (DAQ), 该方法既解决了梯度匹配问题，也解决了量化差异问题，有效提高了各种位宽下的网络性能。

距离感知量化

Distance-aware Quantization

We analyze the effect of quantizing weights and activations of neural
networks on their loss and derive a simple regularization scheme that improves
robustness against post-training quantization. By training quantization-ready
networks, our approach enables storing a single set of weights that can be
quantized on-demand to different bit-widths as energy and memory requirements
of the application change. Unlike quantization-aware training using the
straight-through estimator that only targets a specific bit-width and requires
access to training data and pipeline, our regularization-based method paves the
way for "on the fly'' post-training quantization to various bit-widths. We show
that by modeling quantization as a $\ell_\infty$-bounded perturbation, the
first-order term in the loss expansion can be regularized using the
$\ell_1$-norm of gradients. We experimentally validate the effectiveness of our
regularization scheme on different architectures on CIFAR-10 and ImageNet
datasets.

本文研究了神经网络中权重和激活量化的影响，提出了一种简单的正则化方案来提高其对培训后量化的适应性。通过训练量化 - ready 的网络，我们的方法可以存储一组可按需量化为不同位宽的权重。我们将量化建模为有界扰动，并使用梯度的 L1 范数来对其进行正则化，实验证明了我们该方案的有效性。

用于量化鲁棒性的梯度 L1 正则化

Gradient $\ell_1$ Regularization for Quantization Robustness

Deep neural networks with adaptive configurations have gained increasing
attention due to the instant and flexible deployment of these models on
platforms with different resource budgets. In this paper, we investigate a
novel option to achieve this goal by enabling adaptive bit-widths of weights
and activations in the model. We first examine the benefits and challenges of
training quantized model with adaptive bit-widths, and then experiment with
several approaches including direct adaptation, progressive training and joint
training. We discover that joint training is able to produce comparable
performance on the adaptive model as individual models. We further propose a
new technique named Switchable Clipping Level (S-CL) to further improve
quantized models at the lowest bit-width. With our proposed techniques applied
on a bunch of models including MobileNet-V1/V2 and ResNet-50, we demonstrate
that bit-width of weights and activations is a new option for adaptively
executable deep neural networks, offering a distinct opportunity for improved
accuracy-efficiency trade-off as well as instant adaptation according to the
platform constraints in real-world applications.

本文研究了一种通过使权重和激活的位宽自适应来实现神经网络自适应配置的新方法，并探究了直接适应、渐进适应和联合训练等多种方法，最终提出了一种称为 S-CL 的新技术，证明了通过位宽自适应可以在实现高精度的同时实现高效率，从而在真实世界应用中快速适配。