Recently low-bit (e.g., 8-bit) network quantization has been extensively
studied to accelerate the inference. Besides inference, low-bit training with
quantized gradients can further bring more considerable acceleration, since the
backward process is often computation-intensive. Unfortunately, the
inappropriate quantization of backward propagation usually makes the training
unstable and even crash. There lacks a successful unified low-bit training
framework that can support diverse networks on various tasks. In this paper, we
give an attempt to build a unified 8-bit (INT8) training framework for common
convolutional neural networks from the aspects of both accuracy and speed.
First, we empirically find the four distinctive characteristics of gradients,
which provide us insightful clues for gradient quantization. Then, we
theoretically give an in-depth analysis of the convergence bound and derive two
principles for stable INT8 training. Finally, we propose two universal
techniques, including Direction Sensitive Gradient Clipping that reduces the
direction deviation of gradients and Deviation Counteractive Learning Rate
Scaling that avoids illegal gradient update along the wrong direction. The
experiments show that our unified solution promises accurate and efficient INT8
training for a variety of networks and tasks, including MobileNetV2,
InceptionV3 and object detection that prior studies have never succeeded.
Moreover, it enjoys a strong flexibility to run on off-the-shelf hardware, and
reduces the training time by 22% on Pascal GPU without too much optimization
effort. We believe that this pioneering study will help lead the community
towards a fully unified INT8 training for convolutional neural networks.

本文提出了一个统一的 INT8 训练框架，基于四个梯度特征和两个稳定性原则，以及两种普适性技术，包括方向敏感梯度裁剪和偏移抵消学习率缩放，可以支持各种网络和任务，并在不使用太多优化方法的情况下将训练时间降低了 22％。