We describe the neural-network training framework used in the Kaldi speech
recognition toolkit, which is geared towards training DNNs with large amounts
of training data using multiple GPU-equipped or multi-core machines. In order
to be as hardware-agnostic as possible, we needed a way to use multiple
machines without generating excessive network traffic. Our method is to average
the neural network parameters periodically (typically every minute or two), and
redistribute the averaged parameters to the machines for further training. Each
machine sees different data. By itself, this method does not work very well.
However, we have another method, an approximate and efficient implementation of
Natural Gradient for Stochastic Gradient Descent (NG-SGD), which seems to allow
our periodic-averaging method to work well, as well as substantially improving
the convergence of SGD on a single machine.

本文主要介绍了 Kaldi 语音识别工具包中使用的神经网络训练框架。该框架可用于使用多个带有 GPU 的计算机或多核计算机训练大量训练数据的 DNNs。神经网络参数的定期平均化和分发以及 NG-SGD 的近似高效实现似乎能够使训练方法有效地工作，同时还可以大大提高单个机器上 SGD 的收敛性。