The infinite-width limit of neural networks (NNs) has garnered significant
attention as a theoretical framework for analyzing the behavior of large-scale,
overparametrized networks. By approaching infinite width, NNs effectively
converge to a linear model with features characterized by the neural tangent
kernel (NTK). This establishes a connection between NNs and kernel methods, the
latter of which are well understood. Based on this link, theoretical benefits
and algorithmic improvements have been hypothesized and empirically
demonstrated in synthetic architectures. These advantages include faster
optimization, reliable uncertainty quantification and improved continual
learning. However, current results quantifying the rate of convergence to the
kernel regime suggest that exploiting these benefits requires architectures
that are orders of magnitude wider than they are deep. This assumption raises
concerns that practically relevant architectures do not exhibit behavior as
predicted via the NTK. In this work, we empirically investigate whether the
limiting regime either describes the behavior of large-width architectures used
in practice or is informative for algorithmic improvements. Our empirical
results demonstrate that this is not the case in optimization, uncertainty
quantification or continual learning. This observed disconnect between theory
and practice calls into question the practical relevance of the infinite-width
limit.

通过对无限宽度极限进行实证研究，我们的结果表明：无限宽度极限理论不适用于实际的优化、不确定性量化和持续学习问题，进而对于无限宽度极限的实际相关性提出了质疑。

关于超参数神经网络理论与实践之间的脱节

On the Disconnect Between Theory and Practice of Overparametrized Neural  Networks

These lectures, presented at the 2022 Les Houches Summer School on
Statistical Physics and Machine Learning, focus on the infinite-width limit and
large-width regime of deep neural networks. Topics covered include various
statistical and dynamical properties of these networks. In particular, the
lecturers discuss properties of random deep neural networks; connections
between trained deep neural networks, linear models, kernels, and Gaussian
processes that arise in the infinite-width limit; and perturbative and
non-perturbative treatments of large but finite-width networks, at
initialization and after training.

这篇论文总结了 2022 年 Les Houches 暑期学校上有关深度神经网络无穷宽极限和大宽度区域的讲座，涵盖了这些网络的各种统计和动力学特性。具体来说，讲者讨论了随机深度神经网络的特性、训练后与线性模型、核函数和高斯过程在无穷宽极限中产生的关联，以及初始化和训练后大但有限宽度网络的摄动和非摄动处理。