The infinite-depth paradigm pioneered by Neural ODEs has launched a
renaissance in the search for novel dynamical system-inspired deep learning
primitives; however, their utilization in problems of non-trivial size has
often proved impossible due to poor computational scalability. This work paves
the way for scalable Neural ODEs with time-to-prediction comparable to
traditional discrete networks. We introduce hypersolvers, neural networks
designed to solve ODEs with low overhead and theoretical guarantees on
accuracy. The synergistic combination of hypersolvers and Neural ODEs allows
for cheap inference and unlocks a new frontier for practical application of
continuous-depth models. Experimental evaluations on standard benchmarks, such
as sampling for continuous normalizing flows, reveal consistent pareto
efficiency over classical numerical methods.

本研究介绍了一种名为 Hypersolvers 的神经网络模型，能够以较低的计算成本解决 ODE 问题，在与 Neural ODEs 相结合的情况下，使得基于连续深度模型的实际应用成为可能，实验结果表明 Hypersolve 和 Neural ODE 方法在连续变换计算中的 pareto 效率比传统数字方法更高。

超级求解器：朝着快速的连续深度模型

Hypersolvers: Toward Fast Continuous-Depth Models

We introduce a new family of deep neural network models. Instead of
specifying a discrete sequence of hidden layers, we parameterize the derivative
of the hidden state using a neural network. The output of the network is
computed using a black-box differential equation solver. These continuous-depth
models have constant memory cost, adapt their evaluation strategy to each
input, and can explicitly trade numerical precision for speed. We demonstrate
these properties in continuous-depth residual networks and continuous-time
latent variable models. We also construct continuous normalizing flows, a
generative model that can train by maximum likelihood, without partitioning or
ordering the data dimensions. For training, we show how to scalably
backpropagate through any ODE solver, without access to its internal
operations. This allows end-to-end training of ODEs within larger models.

提出一种新型深度神经网络模型 —— 连续深度模型，其采用了一个神经网络来参数化隐藏状态的导数，并利用黑箱微分方程求解器计算网络输出，使其具有内存成本不变、能够为每个输入自适应地选择评估策略并能显式进行精度 / 速度权衡等特点。研究者进一步证明了通过此模型可以构造出连续正则化流模型，能够通过最大似然进行训练，而不需要对数据维度进行分区或排序，并展示了如何在较大模型内部向任何 ODE 求解器进行可扩展地反向传播，从而实现 ODE 的端到端训练。