Neural network training is inherently sequential where the layers finish the
forward propagation in succession, followed by the calculation and
back-propagation of gradients (based on a loss function) starting from the last
layer. The sequential computations significantly slow down neural network
training, especially the deeper ones. Prediction has been successfully used in
many areas of computer architecture to speed up sequential processing.
Therefore, we propose ADA-GP, that uses gradient prediction adaptively to speed
up deep neural network (DNN) training while maintaining accuracy. ADA-GP works
by incorporating a small neural network to predict gradients for different
layers of a DNN model. ADA-GP uses a novel tensor reorganization to make it
feasible to predict a large number of gradients. ADA-GP alternates between DNN
training using backpropagated gradients and DNN training using predicted
gradients. ADA-GP adaptively adjusts when and for how long gradient prediction
is used to strike a balance between accuracy and performance. Last but not
least, we provide a detailed hardware extension in a typical DNN accelerator to
realize the speed up potential from gradient prediction. Our extensive
experiments with fourteen DNN models show that ADA-GP can achieve an average
speed up of 1.47x with similar or even higher accuracy than the baseline
models. Moreover, it consumes, on average, 34% less energy due to reduced
off-chip memory accesses compared to the baseline hardware accelerator.

提出了一种名为 ADA-GP 的算法，通过使用梯度预测来加速深度神经网络（DNN）的训练，同时保持准确性，并且这种算法具有可适应的特性来平衡精度和性能，同时还提供了一种详细的硬件扩展来实现梯度预测加速潜力。通过对 14 种 DNN 模型的广泛实验表明，与基线模型相比，ADA-GP 可以实现 1.47 倍的平均加速，并且由于减少了与基线硬件加速器相比的外部内存访问，平均消耗的能量降低了 34％。