We propose \emph{Taylorized training} as an initiative towards better
understanding neural network training at finite width. Taylorized training
involves training the $k$-th order Taylor expansion of the neural network at
initialization, and is a principled extension of linearized training---a
recently proposed theory for understanding the success of deep learning.
We experiment with Taylorized training on modern neural network
architectures, and show that Taylorized training (1) agrees with full neural
network training increasingly better as we increase $k$, and (2) can
significantly close the performance gap between linearized and full training.
Compared with linearized training, higher-order training works in more
realistic settings such as standard parameterization and large (initial)
learning rate. We complement our experiments with theoretical results showing
that the approximation error of $k$-th order Taylorized models decay
exponentially over $k$ in wide neural networks.

提出 “Taylorized training” 这一方式以更好地理解有限宽度下神经网络训练的特点，该方法包括在初始状态下培训神经网络的 k 阶泰勒展开，并是线性化训练的有原则的扩展，并通过在现代神经网络结构上尝试 Taylorized training 表明它可以显著缩小线性化和完全训练之间的性能差距，并证明在宽神经网络中，k 阶 Taylorized 模型的逼近误差在 k 中呈指数衰减。