Obtaining theoretical guarantees for neural networks training appears to be a
hard problem in a general case. Recent research has been focused on studying
this problem in the limit of infinite width and two different theories have
been developed: a mean-field (MF) and a constant kernel (NTK) limit theories.
We propose a general framework that provides a link between these seemingly
distinct theories. Our framework out of the box gives rise to a discrete-time
MF limit which was not previously explored in the literature. We prove a
convergence theorem for it and show that it provides a more reasonable
approximation for finite-width nets compared to the NTK limit if learning rates
are not very small. Also, our framework suggests a limit model that coincides
neither with the MF limit nor with the NTK one. We show that for networks with
more than two hidden layers RMSProp training has a non-trivial discrete-time MF
limit but GD training does not have one. Overall, our framework demonstrates
that both MF and NTK limits have considerable limitations in approximating
finite-sized neural nets, indicating the need for designing more accurate
infinite-width approximations for them.

本文提出一个新的框架，将两种不同的神经网络极限理论联系起来，并证明在有限宽度的情况下，离散时间均场极限比常数核极限更有效。