Recurrent neural networks (RNNs) are widely used to model sequential data but their non-linear dependencies between sequence elements prevent parallelizing training over sequence length. We show the training of RNNs with only linear sequential dependencies can be parallelized over the sequence length using the parallel scan algorithm, leading to rapid training on long sequences with small minibatch size. We abstract prior linear sequence models into a new framework of linear surrogate RNNs and develop a linear surrogate long short-term memory (LS-LSTM) powered by a parallel linear recurrence CUDA kernel we implemented. We evaluate the LS-LSTM on a long sequence noisy autoregressive task and find the LS-LSTM achieves slightly superior train and test performance to a similar sized LSTM in 4x less training time. We analyze latency and throughput of the LS-LSTM and find the LS-LSTM reaches up to 175x the throughput of the LSTM in the small minibatch long sequence regime.

该研究论文描述了使用线性顺序依赖关系的RNN可以使用并行扫描算法在序列长度上进行并行化训练，通过开发并行线性递归CUDA内核，加速多种最先进的RNN架构的训练和推理，扩展序列学习到以前无法触及的极长序列区域并成功训练GILR-LSTM进行一百万时间步长的合成序列分类任务。

基于序列长度的线性循环神经网络并行化