Wide neural networks with linear output layer have been shown to be
near-linear, and to have near-constant neural tangent kernel (NTK), in a region
containing the optimization path of gradient descent. These findings seem
counter-intuitive since in general neural networks are highly complex models.
Why does a linear structure emerge when the networks become wide? In this work,
we provide a new perspective on this "transition to linearity" by considering a
neural network as an assembly model recursively built from a set of sub-models
corresponding to individual neurons. In this view, we show that the linearity
of wide neural networks is, in fact, an emerging property of assembling a large
number of diverse "weak" sub-models, none of which dominate the assembly.

该论文探讨了宽神经网络与线性输出层的关系，并发现它们在梯度下降的优化路径区域中几乎是线性的，并且有几乎恒定的神经切向核。通过递归地构建由一组子模型组成的神经网络装配模型的方法，该论文提出了一种新的视角解释宽神经网络的线性结构。

宽神经网络的线性转换是由组装弱模型的新性质

Transition to Linearity of Wide Neural Networks is an Emerging Property  of Assembling Weak Models

Understanding the asymptotic behavior of wide networks is of considerable
interest. In this work, we present a general method for analyzing this large
width behavior. The method is an adaptation of Feynman diagrams, a standard
tool for computing multivariate Gaussian integrals. We apply our method to
study training dynamics, improving existing bounds and deriving new results on
wide network evolution during stochastic gradient descent. Going beyond the
strict large width limit, we present closed-form expressions for higher-order
terms governing wide network training, and test these predictions empirically.

本文介绍了一种基于费曼图的分析大宽度行为的方法，并将其应用于研究网络训练动力学，通过推导新的结果和改进现有界限来推导宽网络演化。

费曼图中宽网络的渐进行为

Asymptotics of Wide Networks from Feynman Diagrams

Wide networks are often believed to have a nice optimization landscape, but
what rigorous results can we prove? To understand the benefit of width, it is
important to identify the difference between wide and narrow networks. In this
work, we prove that from narrow to wide networks, there is a phase transition
from having sub-optimal basins to no sub-optimal basins. Specifically, we prove
two results: on the positive side, for any continuous activation functions, the
loss surface of a class of wide networks has no sub-optimal basins, where
"basin" is defined as the set-wise strict local minimum; on the negative side,
for a large class of networks with width below a threshold, we construct strict
local minima that are not global. These two results together show the phase
transition from narrow to wide networks.

本文证明了从窄网络到宽网络的过渡存在一个相变，窄网络存在次优盆地（sub-optimal basins），而宽网络不存在。具体地，采用连续激活函数的一类宽网络损失曲面没有亚优盆地，但是对于宽度低于阈值的大类网络，存在不是全局最小值而是严格局部最小值的情况。