In this work we apply model averaging to parallel training of deep neural
network (DNN). Parallelization is done in a model averaging manner. Data is
partitioned and distributed to different nodes for local model updates, and
model averaging across nodes is done every few minibatches. We use multiple
GPUs for data parallelization, and Message Passing Interface (MPI) for
communication between nodes, which allows us to perform model averaging
frequently without losing much time on communication. We investigate the
effectiveness of Natural Gradient Stochastic Gradient Descent (NG-SGD) and
Restricted Boltzmann Machine (RBM) pretraining for parallel training in
model-averaging framework, and explore the best setups in term of different
learning rate schedules, averaging frequencies and minibatch sizes. It is shown
that NG-SGD and RBM pretraining benefits parameter-averaging based model
training. On the 300h Switchboard dataset, a 9.3 times speedup is achieved
using 16 GPUs and 17 times speedup using 32 GPUs with limited decoding accuracy
loss.

本研究针对深度神经网络（DNN）的并行训练使用模型平均方法。多 GPU 数据并行化，MPI 进行节点间通信，每隔几个 minibatches 进行模型平均。针对不同的学习速度、平均频率和 minibatch 尺寸探索最佳设置，研究发现 “NG-SGD” 和 “RBM” 预训练有利于基于参数平均的模型训练，可以实现 9.3 倍和 17 倍的加速，只有少量解码精度的损失。