Finishing 90-epoch ImageNet-1k training with ResNet-50 on a NVIDIA M40 GPU
takes 14 days. This training requires 10^18 single precision operations in
total. On the other hand, the world's current fastest supercomputer can finish
2 * 10^17 single precision operations per second (Dongarra et al 2017,
this https URL). If we can make full use of the
supercomputer for DNN training, we should be able to finish the 90-epoch
ResNet-50 training in one minute. However, the current bottleneck for fast DNN
training is in the algorithm level. Specifically, the current batch size (e.g.
512) is too small to make efficient use of many processors. For large-scale DNN
training, we focus on using large-batch data-parallelism synchronous SGD
without losing accuracy in the fixed epochs. The LARS algorithm (You, Gitman,
Ginsburg, 2017, arXiv:1708.03888) enables us to scale the batch size to
extremely large case (e.g. 32K). We finish the 100-epoch ImageNet training with
AlexNet in 11 minutes on 1024 CPUs. About three times faster than Facebook's
result (Goyal et al 2017, arXiv:1706.02677), we finish the 90-epoch ImageNet
training with ResNet-50 in 20 minutes on 2048 KNLs without losing accuracy.
State-of-the-art ImageNet training speed with ResNet-50 is 74.9% top-1 test
accuracy in 15 minutes. We got 74.9% top-1 test accuracy in 64 epochs, which
only needs 14 minutes. Furthermore, when we increase the batch size to above
16K, our accuracy is much higher than Facebook's on corresponding batch sizes.
Our source code is available upon request.

采用大批量数据并行同步 SGD 的 LARS 算法，我们在 2048 个 KNL 上完成了 90 个周期 ResNet-50 的 ImageNet 训练，仅需 20 分钟，同时达到 74.9% 的测试精度。

分钟级 ImageNet 训练

ImageNet Training in Minutes

In recent years, deep neural networks (DNN) have demonstrated significant
business impact in large scale analysis and classification tasks such as speech
recognition, visual object detection, pattern extraction, etc. Training of
large DNNs, however, is universally considered as time consuming and
computationally intensive task that demands datacenter-scale computational
resources recruited for many days. Here we propose a concept of resistive
processing unit (RPU) devices that can potentially accelerate DNN training by
orders of magnitude while using much less power. The proposed RPU device can
store and update the weight values locally thus minimizing data movement during
training and allowing to fully exploit the locality and the parallelism of the
training algorithm. We identify the RPU device and system specifications for
implementation of an accelerator chip for DNN training in a realistic
CMOS-compatible technology. For large DNNs with about 1 billion weights this
massively parallel RPU architecture can achieve acceleration factors of 30,000X
compared to state-of-the-art microprocessors while providing power efficiency
of 84,000 GigaOps/s/W. Problems that currently require days of training on a
datacenter-size cluster with thousands of machines can be addressed within
hours on a single RPU accelerator. A system consisted of a cluster of RPU
accelerators will be able to tackle Big Data problems with trillions of
parameters that is impossible to address today like, for example, natural
speech recognition and translation between all world languages, real-time
analytics on large streams of business and scientific data, integration and
analysis of multimodal sensory data flows from massive number of IoT (Internet
of Things) sensors.

本篇论文提出了利用抵抗处理单元（RPU）设备加速深度神经网络（DNN）训练的概念，通过使用 RPU 设备，可以在更短的时间内，使用更少的功耗来完成大规模 DNN 的训练，并且可以实现比现有技术高达 30000 倍的加速因子和 84000 GigaOps /s/ W 的功率效率。