Learning with recurrent neural networks (RNNs) on long sequences is a
notoriously difficult task. There are three major challenges: 1) complex
dependencies, 2) vanishing and exploding gradients, and 3) efficient
parallelization. In this paper, we introduce a simple yet effective RNN
connection structure, the DilatedRNN, which simultaneously tackles all of these
challenges. The proposed architecture is characterized by multi-resolution
dilated recurrent skip connections and can be combined flexibly with diverse
RNN cells. Moreover, the DilatedRNN reduces the number of parameters needed and
enhances training efficiency significantly, while matching state-of-the-art
performance (even with standard RNN cells) in tasks involving very long-term
dependencies. To provide a theory-based quantification of the architecture's
advantages, we introduce a memory capacity measure, the mean recurrent length,
which is more suitable for RNNs with long skip connections than existing
measures. We rigorously prove the advantages of the DilatedRNN over other
recurrent neural architectures. The code for our method is publicly available
at this https URL

本研究提出了一种称为 DilatedRNN 的简单且有效的循环神经网络连接结构，它可以同时解决 RNN 在处理长序列时遇到的复杂依赖、梯度消失和爆炸的问题，还能够提高训练效率并匹配最先进技术，并且引入了一种适用于具有长跳跃连接的 RNN 的内存容量度量，证明了 DilatedRNN 相对于其他循环神经网络结构的优点。