Neural ordinary differential equations (NODEs) treat computation of
intermediate feature vectors as trajectories of ordinary differential equation
parameterized by a neural network. In this paper, we propose a novel model,
delay differential neural networks (DDNN), inspired by delay differential
equations (DDEs). The proposed model considers the derivative of the hidden
feature vector as a function of the current feature vector and past feature
vectors (history). The function is modelled as a neural network and
consequently, it leads to continuous depth alternatives to many recent ResNet
variants. We propose two different DDNN architectures, depending on the way
current and past feature vectors are considered. For training DDNNs, we provide
a memory-efficient adjoint method for computing gradients and back-propagate
through the network. DDNN improves the data efficiency of NODE by further
reducing the number of parameters without affecting the generalization
performance. Experiments conducted on synthetic and real-world image
classification datasets such as Cifar10 and Cifar100 show the effectiveness of
the proposed models.

本文介绍了基于延迟微分方程（DDEs）的一种新型模型延迟微分神经网络（DDNN），其包括两种不同的架构。通过提供一种内存高效的方法计算梯度并反向传播至网络中，DDNN 显著减少了一些最近 ResNet 变体的参数数量并提高了图像分类数据集，例如 Cifar10 和 Cifar100 的效果。