Implicit layer deep learning techniques, like Neural Differential Equations,
have become an important modeling framework due to their ability to adapt to
new problems automatically. Training a neural differential equation is
effectively a search over a space of plausible dynamical systems. However,
controlling the computational cost for these models is difficult since it
relies on the number of steps the adaptive solver takes. Most prior works have
used higher-order methods to reduce prediction timings while greatly increasing
training time or reducing both training and prediction timings by relying on
specific training algorithms, which are harder to use as a drop-in replacement
due to strict requirements on automatic differentiation. In this manuscript, we
use internal cost heuristics of adaptive differential equation solvers at
stochastic time points to guide the training toward learning a dynamical system
that is easier to integrate. We "close the black-box" and allow the use of our
method with any adjoint technique for gradient calculations of the differential
equation solution. We perform experimental studies to compare our method to
global regularization to show that we attain similar performance numbers
without compromising the flexibility of implementation on ordinary differential
equations (ODEs) and stochastic differential equations (SDEs). We develop two
sampling strategies to trade off between performance and training time. Our
method reduces the number of function evaluations to 0.556-0.733x and
accelerates predictions by 1.3-2x.

通过使用内部成本启发式算法，本文开发了两种采样策略来减少函数评估数量并加速预测，与全局正则化相比，我们的方法在普通微分方程和随机微分方程中具有相似的性能而不会影响实施的灵活性。