Recent theoretical work has established connections between over-parametrized
neural networks and linearized models governed by he Neural Tangent Kernels
(NTKs). NTK theory leads to concrete convergence and generalization results,
yet the empirical performance of neural networks are observed to exceed their
linearized models, suggesting insufficiency of this theory.
Towards closing this gap, we investigate the training of over-parametrized
neural networks that are beyond the NTK regime yet still governed by the Taylor
expansion of the network. We bring forward the idea of \emph{randomizing} the
neural networks, which allows them to escape their NTK and couple with
quadratic models. We show that the optimization landscape of randomized
two-layer networks are nice and amenable to escaping-saddle algorithms. We
prove concrete generalization and expressivity results on these randomized
networks, which lead to sample complexity bounds (of learning certain simple
functions) that match the NTK and can in addition be better by a dimension
factor when mild distributional assumptions are present. We demonstrate that
our randomization technique can be generalized systematically beyond the
quadratic case, by using it to find networks that are coupled with higher-order
terms in their Taylor series.

该研究探讨了超参数化神经网络的训练，通过随机化神经网络，使其模型不再受限于 NTK，允许其与二次模型进行耦合，并且证明了产生的随机网络的优化具有良好的局面性态，且样本复杂度与 NTK 相匹配，同时在分布特定条件下可更为优秀。