New hardware can substantially increase the speed and efficiency of deep
neural network training. To guide the development of future hardware
architectures, it is pertinent to explore the hardware and machine learning
properties of alternative training algorithms. In this work we evaluate the use
of small batch, fine-grained Pipelined Backpropagation, an asynchronous
pipeline parallel training algorithm that has significant hardware advantages.
We introduce two methods, Spike Compensation and Linear Weight Prediction, that
effectively mitigate the downsides caused by the asynchronicity of Pipelined
Backpropagation and outperform existing techniques in our setting. We show that
appropriate normalization and small batch sizes can also aid training. With our
methods, fine-grained Pipelined Backpropagation using a batch size of one can
match the accuracy of SGD for multiple networks trained on CIFAR-10 and
ImageNet. Simple scaling rules allow the use of existing hyperparameters for
traditional training without additional tuning.

本文研究了深度神经网络的硬件加速器，并提出了一种具有硬件优势的异步管道并行训练算法。通过引入 Spike Compensation 和 Linear Weight Prediction 两种方法，它有效地减轻了由 Pipelined Backpropagation 的异步性造成的缺点，并优于现有技术。适当的标准化和小批量大小也有助于训练，与 SGD 相比，它能够在 CIFAR-10 和 ImageNet 上为多个网络的训练匹配准确度。