Recent breakthroughs in deep neural networks (DNNs) have fueled a tremendous
demand for intelligent edge devices featuring on-site learning, while the
practical realization of such systems remains a challenge due to the limited
resources available at the edge and the required massive training costs for
state-of-the-art (SOTA) DNNs. As reducing precision is one of the most
effective knobs for boosting training time/energy efficiency, there has been a
growing interest in low-precision DNN training. In this paper, we explore from
an orthogonal direction: how to fractionally squeeze out more training cost
savings from the most redundant bit level, progressively along the training
trajectory and dynamically per input. Specifically, we propose FracTrain that
integrates (i) progressive fractional quantization which gradually increases
the precision of activations, weights, and gradients that will not reach the
precision of SOTA static quantized DNN training until the final training stage,
and (ii) dynamic fractional quantization which assigns precisions to both the
activations and gradients of each layer in an input-adaptive manner, for only
"fractionally" updating layer parameters. Extensive simulations and ablation
studies (six models, four datasets, and three training settings including
standard, adaptation, and fine-tuning) validate the effectiveness of FracTrain
in reducing computational cost and hardware-quantified energy/latency of DNN
training while achieving a comparable or better (-0.12%~+1.87%) accuracy. For
example, when training ResNet-74 on CIFAR-10, FracTrain achieves 77.6% and
53.5% computational cost and training latency savings, respectively, compared
with the best SOTA baseline, while achieving a comparable (-0.07%) accuracy.
Our codes are available at: this https URL

本研究提出了一种名为 FracTrain 的系统，通过渐进式的分数量化和动态分数量化的方式来降低神经网络训练的计算成本和硬件开销，同时实现可比拟或更高的准确率。