Recent work has attempted to interpret residual networks (ResNets) as one
step of a forward Euler discretization of an ordinary differential equation,
focusing mainly on syntactic algebraic similarities between the two systems.
Discrete dynamical integrators of continuous dynamical systems, however, have a
much richer structure. We first show that ResNets fail to be meaningful
dynamical integrators in this richer sense. We then demonstrate that neural
network models can learn to represent continuous dynamical systems, with this
richer structure and properties, by embedding them into higher-order numerical
integration schemes, such as the Runge Kutta schemes. Based on these insights,
we introduce ContinuousNet as a continuous-in-depth generalization of ResNet
architectures. ContinuousNets exhibit an invariance to the particular
computational graph manifestation. That is, the continuous-in-depth model can
be evaluated with different discrete time step sizes, which changes the number
of layers, and different numerical integration schemes, which changes the graph
connectivity. We show that this can be used to develop an incremental-in-depth
training scheme that improves model quality, while significantly decreasing
training time. We also show that, once trained, the number of units in the
computational graph can even be decreased, for faster inference with
little-to-no accuracy drop.

通过将连续动力学系统嵌入高阶数值积分方案中，我们构造了一种连续的深度神经网络框架 ——ContinuousNet，其具有与时间步长大小无关的层次独立性和增量深度训练方法，实现了在保持准确性的同时加快推断速度。