Due to their dynamic properties such as irregular sampling rate and
high-frequency sampling, Continuous Time Series (CTS) are found in many
applications. Since CTS with irregular sampling rate are difficult to model
with standard Recurrent Neural Networks (RNNs), RNNs have been generalised to
have continuous-time hidden dynamics defined by a Neural Ordinary Differential
Equation (Neural ODE), leading to the ODE-RNN model. Another approach that
provides a better modelling is that of the Latent ODE model, which constructs a
continuous-time model where a latent state is defined at all times. The Latent
ODE model uses a standard RNN as the encoder and a Neural ODE as the decoder.
However, since the RNN encoder leads to difficulties with missing data and
ill-defined latent variables, a Latent ODE-RNN model has recently been proposed
that uses a ODE-RNN model as the encoder instead. Both the Latent ODE and
Latent ODE-RNN models are difficult to train due to the vanishing and exploding
gradients problem. To overcome this problem, the main contribution of this
paper is to propose and illustrate a new model based on a new Latent ODE using
an ODE-LSTM (Long Short-Term Memory) network as an encoder -- the Latent
ODE-LSTM model. To limit the growth of the gradients the Norm Gradient Clipping
strategy was embedded on the Latent ODE-LSTM model. The performance evaluation
of the new Latent ODE-LSTM (with and without Norm Gradient Clipping) for
modelling CTS with regular and irregular sampling rates is then demonstrated.
Numerical experiments show that the new Latent ODE-LSTM performs better than
Latent ODE-RNNs and can avoid the vanishing and exploding gradients during
training.

提出了一种基于 ODE-LSTM 网络的新型隐状态 ODE 模型，用于建模连续时间序列数据，实验结果表明其相较于 Latent ODE-RNN 模型具有更好的性能且能够避免训练过程中出现的梯度消失和梯度爆炸问题。

利用潜变量 ODE-LSTM 方法增强连续时间序列建模

Enhancing Continuous Time Series Modelling with a Latent ODE-LSTM  Approach

Neural differential equations are a promising new member in the neural
network family. They show the potential of differential equations for time
series data analysis. In this paper, the strength of the ordinary differential
equation (ODE) is explored with a new extension. The main goal of this work is
to answer the following questions: (i)~can ODE be used to redefine the existing
neural network model? (ii)~can Neural ODEs solve the irregular sampling rate
challenge of existing neural network models for a continuous time series, i.e.,
length and dynamic nature, (iii)~how to reduce the training and evaluation time
of existing Neural ODE systems? This work leverages the mathematical foundation
of ODEs to redesign traditional RNNs such as Long Short-Term Memory (LSTM) and
Gated Recurrent Unit (GRU). The main contribution of this paper is to
illustrate the design of two new ODE-based RNN models (GRU-ODE model and
LSTM-ODE) which can compute the hidden state and cell state at any point of
time using an ODE solver. These models reduce the computation overhead of
hidden state and cell state by a vast amount. The performance evaluation of
these two new models for learning continuous time series with irregular
sampling rate is then demonstrated. Experiments show that these new ODE based
RNN models require less training time than Latent ODEs and conventional Neural
ODEs. They can achieve higher accuracy quickly, and the design of the neural
network is simpler than, previous neural ODE systems.

该研究论文介绍了一种使用 ODE 的时间序列数据分析方法，提出基于 ODE 的 RNN 模型，可在较短的训练时间内学习具有不规则采样率的连续时间序列，并且计算效率更高、精度更高、设计更简单。