Recurrent neural networks (RNNs) with continuous-time hidden states are a
natural fit for modeling irregularly-sampled time series. These models,
however, face difficulties when the input data possess long-term dependencies.
We prove that similar to standard RNNs, the underlying reason for this issue is
the vanishing or exploding of the gradient during training. This phenomenon is
expressed by the ordinary differential equation (ODE) representation of the
hidden state, regardless of the ODE solver's choice. We provide a solution by
designing a new algorithm based on the long short-term memory (LSTM) that
separates its memory from its time-continuous state. This way, we encode a
continuous-time dynamical flow within the RNN, allowing it to respond to inputs
arriving at arbitrary time-lags while ensuring a constant error propagation
through the memory path. We call these RNN models ODE-LSTMs. We experimentally
show that ODE-LSTMs outperform advanced RNN-based counterparts on non-uniformly
sampled data with long-term dependencies. All code and data is available at
this https URL

我们提出了一种基于长短期记忆机制的算法来解决非均匀采样数据上具有长期依赖性的序列建模问题，其核心是将记忆从时间连续状态中分离出来，形成了一种新的 ODE-LSTM 模型。实验证明，该模型在非均匀采样数据的长期依赖建模上优于基于其他递归神经网络的模型。