Neural networks with wide layers have attracted significant attention due to
their equivalence to Gaussian processes, enabling perfect fitting of training
data while maintaining generalization performance, known as benign overfitting.
However, existing results mainly focus on shallow or finite-depth networks,
necessitating a comprehensive analysis of wide neural networks with
infinite-depth layers, such as neural ordinary differential equations (ODEs)
and deep equilibrium models (DEQs). In this paper, we specifically investigate
the deep equilibrium model (DEQ), an infinite-depth neural network with shared
weight matrices across layers. Our analysis reveals that as the width of DEQ
layers approaches infinity, it converges to a Gaussian process, establishing
what is known as the Neural Network and Gaussian Process (NNGP) correspondence.
Remarkably, this convergence holds even when the limits of depth and width are
interchanged, which is not observed in typical infinite-depth Multilayer
Perceptron (MLP) networks. Furthermore, we demonstrate that the associated
Gaussian vector remains non-degenerate for any pairwise distinct input data,
ensuring a strictly positive smallest eigenvalue of the corresponding kernel
matrix using the NNGP kernel. These findings serve as fundamental elements for
studying the training and generalization of DEQs, laying the groundwork for
future research in this area.

当 DEQ 层的宽度趋近于无穷大时，它收敛到一个高斯过程，建立了所谓的神经网络和高斯过程 (NNGP) 的对应关系，这种收敛甚至在深度和宽度的极限互换的情况下也成立，这些发现为研究 DEQ 的训练和泛化提供了基础，为未来的研究奠定了基础。

宽神经网络作为高斯过程：深度平衡模型的教训

Wide Neural Networks as Gaussian Processes: Lessons from Deep  Equilibrium Models

We prove that for an $L$-layer fully-connected linear neural network, if the
width of every hidden layer is $\tilde\Omega (L \cdot r \cdot d_{\mathrm{out}}
\cdot \kappa^3 )$, where $r$ and $\kappa$ are the rank and the condition number
of the input data, and $d_{\mathrm{out}}$ is the output dimension, then
gradient descent with Gaussian random initialization converges to a global
minimum at a linear rate. The number of iterations to find an
$\epsilon$-suboptimal solution is $O(\kappa \log(\frac{1}{\epsilon}))$. Our
polynomial upper bound on the total running time for wide deep linear networks
and the $\exp\left(\Omega\left(L\right)\right)$ lower bound for narrow deep
linear neural networks [Shamir, 2018] together demonstrate that wide layers are
necessary for optimizing deep models.

证明全连接线性神经网络每个隐藏层的宽度大于 $\tilde\Omega (L \cdot r \cdot d_{\mathrm {out}} \cdot \kappa^3)$ 时， 高斯随机初始化的梯度下降算法会以线性速率收敛到全局最小值，但窄层深度线性神经网络需要 $\exp\left (\Omega\left (L\right)\right)$ 操作时间，这表明宽层对于优化深度模型是必要的。