The design of recurrent neural networks (RNNs) to accurately process
sequential inputs with long-time dependencies is very challenging on account of
the exploding and vanishing gradient problem. To overcome this, we propose a
novel RNN architecture which is based on a structure preserving discretization
of a Hamiltonian system of second-order ordinary differential equations that
models networks of oscillators. The resulting RNN is fast, invertible (in
time), memory efficient and we derive rigorous bounds on the hidden state
gradients to prove the mitigation of the exploding and vanishing gradient
problem. A suite of experiments are presented to demonstrate that the proposed
RNN provides state of the art performance on a variety of learning tasks with
(very) long-time dependencies.

本文提出了一种基于哈密顿系统的离散化的循环神经网络架构，解决长时依赖序列输入处理的梯度消失和爆炸问题，实验表明该方法在各种学习任务中提供了最先进的性能。

UnICORNN: 用于学习非常长时间依赖关系的循环模型

UnICORNN: A recurrent model for learning very long time dependencies

Recurrent neural networks are known for their notorious exploding and
vanishing gradient problem (EVGP). This problem becomes more evident in tasks
where the information needed to correctly solve them exist over long time
scales, because EVGP prevents important gradient components from being
back-propagated adequately over a large number of steps. We introduce a simple
stochastic algorithm (\textit{h}-detach) that is specific to LSTM optimization
and targeted towards addressing this problem. Specifically, we show that when
the LSTM weights are large, the gradient components through the linear path
(cell state) in the LSTM computational graph get suppressed. Based on the
hypothesis that these components carry information about long term dependencies
(which we show empirically), their suppression can prevent LSTMs from capturing
them. Our algorithm\footnote{Our code is available at
this https URL} prevents gradients flowing through
this path from getting suppressed, thus allowing the LSTM to capture such
dependencies better. We show significant improvements over vanilla LSTM
gradient based training in terms of convergence speed, robustness to seed and
learning rate, and generalization using our modification of LSTM gradient on
various benchmark datasets.

本文介绍了一种针对长期依赖问题的 LSTM 优化方案，通过采用一种简单的随机算法（h-detach）可以避免梯度消失问题，通过提升梯度在计算图中的表现，进而提高模型优化的鲁棒性效果，使 LSTM 捕获更好的长期依赖关系并在多个基准数据集上显示出显着的性能提升。