In this paper, we present GossipGraD - a gossip communication protocol based
Stochastic Gradient Descent (SGD) algorithm for scaling Deep Learning (DL)
algorithms on large-scale systems. The salient features of GossipGraD are: 1)
reduction in overall communication complexity from {\Theta}(log(p)) for p
compute nodes in well-studied SGD to O(1), 2) model diffusion such that compute
nodes exchange their updates (gradients) indirectly after every log(p) steps,
3) rotation of communication partners for facilitating direct diffusion of
gradients, 4) asynchronous distributed shuffle of samples during the
feedforward phase in SGD to prevent over-fitting, 5) asynchronous communication
of gradients for further reducing the communication cost of SGD and GossipGraD.
We implement GossipGraD for GPU and CPU clusters and use NVIDIA GPUs (Pascal
P100) connected with InfiniBand, and Intel Knights Landing (KNL) connected with
Aries network. We evaluate GossipGraD using well-studied dataset ImageNet-1K
(~250GB), and widely studied neural network topologies such as GoogLeNet and
ResNet50 (current winner of ImageNet Large Scale Visualization Research
Challenge (ILSVRC)). Our performance evaluation using both KNL and Pascal GPUs
indicates that GossipGraD can achieve perfect efficiency for these datasets and
their associated neural network topologies. Specifically, for ResNet50,
GossipGraD is able to achieve ~100% compute efficiency using 128 NVIDIA Pascal
P100 GPUs - while matching the top-1 classification accuracy published in
literature.

本研究提出了基于疏泊梯度下降算法的 GossipGraD 聊天协议，用于大规模系统的深度学习算法。通过优化 GossipGraD 的五个显著特征，最终实现对 ImageNet-1K 数据集的高效计算和分类。