The problem of quantizing the activations of a deep neural network is
considered. An examination of the popular binary quantization approach shows
that this consists of approximating a classical non-linearity, the hyperbolic
tangent, by two functions: a piecewise constant sign function, which is used in
feedforward network computations, and a piecewise linear hard tanh function,
used in the backpropagation step during network learning. The problem of
approximating the ReLU non-linearity, widely used in the recent deep learning
literature, is then considered. An half-wave Gaussian quantizer (HWGQ) is
proposed for forward approximation and shown to have efficient implementation,
by exploiting the statistics of of network activations and batch normalization
operations commonly used in the literature. To overcome the problem of gradient
mismatch, due to the use of different forward and backward approximations,
several piece-wise backward approximators are then investigated. The
implementation of the resulting quantized network, denoted as HWGQ-Net, is
shown to achieve much closer performance to full precision networks, such as
AlexNet, ResNet, GoogLeNet and VGG-Net, than previously available low-precision
networks, with 1-bit binary weights and 2-bit quantized activations.

研究深度神经网络的激活量化问题，提出了一种半波高斯量化器（HWGQ）来近似 ReLU 非线性激活函数，并探讨了多种反向逼近方式以解决梯度失配问题，实现的量化网络 HWGQ-Net 的性能比以前的低精度网络如二进制权重和二位量化激活的网络接近全精度网络（如 AlexNet，ResNet，GoogLeNet 和 VGG-Net）。