Neural SDEs combine many of the best qualities of both RNNs and SDEs: memory
efficient training, high-capacity function approximation, and strong priors on
model space. This makes them a natural choice for modelling many types of
temporal dynamics. Training a Neural SDE (either as a VAE or as a GAN) requires
backpropagating through an SDE solve. This may be done by solving a
backwards-in-time SDE whose solution is the desired parameter gradients.
However, this has previously suffered from severe speed and accuracy issues,
due to high computational cost and numerical truncation errors. Here, we
overcome these issues through several technical innovations. First, we
introduce the \textit{reversible Heun method}. This is a new SDE solver that is
\textit{algebraically reversible}: eliminating numerical gradient errors, and
the first such solver of which we are aware. Moreover it requires half as many
function evaluations as comparable solvers, giving up to a $1.98\times$
speedup. Second, we introduce the \textit{Brownian Interval}: a new, fast,
memory efficient, and exact way of sampling \textit{and reconstructing}
Brownian motion. With this we obtain up to a $10.6\times$ speed improvement
over previous techniques, which in contrast are both approximate and relatively
slow. Third, when specifically training Neural SDEs as GANs (Kidger et al.
2021), we demonstrate how SDE-GANs may be trained through careful weight
clipping and choice of activation function. This reduces computational cost
(giving up to a $1.87\times$ speedup) and removes the numerical truncation
errors associated with gradient penalty. Altogether, we outperform the
state-of-the-art by substantial margins, with respect to training speed, and
with respect to classification, prediction, and MMD test metrics. We have
contributed implementations of all of our techniques to the torchsde library to
help facilitate their adoption.

本文介绍了利用可逆且代数上可逆的 Heun 方法以及优化的布朗运动采样技术，加速求解反向 SDE 以训练神经 SDE 模型，并提出了用于 GAN 的训练技巧，实验表明我们的算法在速度和预测效果上超过了现有技术。