Efficient deep neural network (DNN) inference on mobile or embedded devices
typically involves quantization of the network parameters and activations. In
particular, mixed precision networks achieve better performance than networks
with homogeneous bitwidth for the same size constraint. Since choosing the
optimal bitwidths is not straight forward, training methods, which can learn
them, are desirable. Differentiable quantization with straight-through
gradients allows to learn the quantizer's parameters using gradient methods. We
show that a suited parametrization of the quantizer is the key to achieve a
stable training and a good final performance. Specifically, we propose to
parametrize the quantizer with the step size and dynamic range. The bitwidth
can then be inferred from them. Other parametrizations, which explicitly use
the bitwidth, consistently perform worse. We confirm our findings with
experiments on CIFAR-10 and ImageNet and we obtain mixed precision DNNs with
learned quantization parameters, achieving state-of-the-art performance.

本文提出一种不同的量化方法，使用不同的参数化方法来训练深度神经网络，从而达到更高效深度神经网络推理的效果。通过多组实验验证，使用该方法训练得到的量化参数可以达到最佳效果。

混合精度 DNNs：只需良好参数化

Mixed Precision DNNs: All you need is a good parametrization

Neural network quantization has become an important research area due to its
great impact on deployment of large models on resource constrained devices. In
order to train networks that can be effectively discretized without loss of
performance, we introduce a differentiable quantization procedure.
Differentiability can be achieved by transforming continuous distributions over
the weights and activations of the network to categorical distributions over
the quantization grid. These are subsequently relaxed to continuous surrogates
that can allow for efficient gradient-based optimization. We further show that
stochastic rounding can be seen as a special case of the proposed approach and
that under this formulation the quantization grid itself can also be optimized
with gradient descent. We experimentally validate the performance of our method
on MNIST, CIFAR 10 and Imagenet classification.

本文介绍一种可微的量化过程，将连续分布转换为分类型分布，然后通过松弛到连续的代理来实现高效的基于梯度的优化，还展示了随机舍入可理解为所提出方法的一个特例。通过实验证明本方法对于 MNIST、CIFAR 10 和 Imagenet 等各种问题的分类都具有很好的表现.