An efficient algorithm for recurrent neural network training is presented.
The approach increases the training speed for tasks where a length of the input
sequence may vary significantly. The proposed approach is based on the optimal
batch bucketing by input sequence length and data parallelization on multiple
graphical processing units. The baseline training performance without sequence
bucketing is compared with the proposed solution for a different number of
buckets. An example is given for the online handwriting recognition task using
an LSTM recurrent neural network. The evaluation is performed in terms of the
wall clock time, number of epochs, and validation loss value.

本研究提出了一种基于最佳批处理按输入序列长度和数据并行化的高效循环神经网络训练算法，以在线手写识别任务为例，通过比较不同数量桶的基准训练性能和提出的解决方案，考察了墙钟时间、纪元数和验证损失值等方面的评估结果。

利用序列桶技术和多 GPU 数据并行化加速循环神经网络的训练

Accelerating recurrent neural network training using sequence bucketing  and multi-GPU data parallelization

There is significant recent interest to parallelize deep learning algorithms
in order to handle the enormous growth in data and model sizes. While most
advances focus on model parallelization and engaging multiple computing agents
via using a central parameter server, aspect of data parallelization along with
decentralized computation has not been explored sufficiently. In this context,
this paper presents a new consensus-based distributed SGD (CDSGD) (and its
momentum variant, CDMSGD) algorithm for collaborative deep learning over fixed
topology networks that enables data parallelization as well as decentralized
computation. Such a framework can be extremely useful for learning agents with
access to only local/private data in a communication constrained environment.
We analyze the convergence properties of the proposed algorithm with strongly
convex and nonconvex objective functions with fixed and diminishing step sizes
using concepts of Lyapunov function construction. We demonstrate the efficacy
of our algorithms in comparison with the baseline centralized SGD and the
recently proposed federated averaging algorithm (that also enables data
parallelism) based on benchmark datasets such as MNIST, CIFAR-10 and CIFAR-100.

本研究提出一种基于共识的分布式 SGD 算法，并使用 Lyapunov 方法分析其强凸和非凸目标函数的收敛性，能够支持数据并行和去中心化计算，适用于具有通信限制的本地数据的学习代理，实验表明该算法在 MNIST、CIFAR-10 和 CIFAR-100 数据集上，能够显著优于集中式 SGD 和联合平均算法。