Deep neural network (DNN) quantization converting floating-point (FP) data in
the network to integers (INT) is an effective way to shrink the model size for
memory saving and simplify the operations for compute acceleration. Recently,
researches on DNN quantization develop from inference to training, laying a
foundation for the online training on accelerators. However, existing schemes
leaving batch normalization (BN) untouched during training are mostly
incomplete quantization that still adopts high precision FP in some parts of
the data paths. Currently, there is no solution that can use only low bit-width
INT data during the whole training process of large-scale DNNs with acceptable
accuracy. In this work, through decomposing all the computation steps in DNNs
and fusing three special quantization functions to satisfy the different
precision requirements, we propose a unified complete quantization framework
termed as ``WAGEUBN'' to quantize DNNs involving all data paths including W
(Weights), A (Activation), G (Gradient), E (Error), U (Update), and BN.
Moreover, the Momentum optimizer is also quantized to realize a completely
quantized framework. Experiments on ResNet18/34/50 models demonstrate that
WAGEUBN can achieve competitive accuracy on the ImageNet dataset. For the first
time, the study of quantization in large-scale DNNs is advanced to the full
8-bit INT level. In this way, all the operations in the training and inference
can be bit-wise operations, pushing towards faster processing speed, decreased
memory cost, and higher energy efficiency. Our throughout quantization
framework has great potential for future efficient portable devices with online
learning ability.

本文提出了一种名为 WAGEUBN 的统一完整量化框架，可将神经网络的数据路径（包括所有数据类型 W，A，G，E，U 和 BN）中所有数据转换为低位整数以实现全面量化和在线训练，并在 ImageNet 数据集上获得了可比较的精度，展示了在大规模 DNNs 中 8 位 INT 级别的完整量化的功能性。