During the last few years, significant attention has been paid to the stochastic training of artificial neural networks, which is known as an effective regularization approach that helps improve the generalization capability of trained models. In this work, the method of modified equations is applied to show that the residual network and its variants with noise injection can be regarded as weak approximations of stochastic differential equations. Such observations enable us to bridge the stochastic training processes with the optimal control of backward Kolmogorov's equations. This not only offers a novel perspective on the effects of regularization from the loss landscape viewpoint but also sheds light on the design of more reliable and efficient stochastic training strategies. As an example, we propose a new way to utilize Bernoulli dropout within the plain residual network architecture and conduct experiments on a real-world image classification task to substantiate our theoretical findings.

本文研究了基于改进方程的方法，表明残差网络及其变体可以被视为弱逼近随机微分方程。从损失景观的角度提供了关于正则化效应的新视角，并为设计更可靠和高效的随机训练策略提供了启示。我们提出了一种利用伯努利丢弃来进行实验的新方法，从而验证了我们的理论发现。

残余网络的随机训练：微分方程视角