This paper presents a succinct derivation of the training and generalization
performance of a variety of high-dimensional ridge regression models using the
basic tools of random matrix theory and free probability. We provide an
introduction and review of recent results on these topics, aimed at readers
with backgrounds in physics and deep learning. Analytic formulas for the
training and generalization errors are obtained in a few lines of algebra
directly from the properties of the $S$-transform of free probability. This
allows for a straightforward identification of the sources of power-law scaling
in model performance. We compute the generalization error of a broad class of
random feature models. We find that in all models, the $S$-transform
corresponds to the train-test generalization gap, and yields an analogue of the
generalized-cross-validation estimator. Using these techniques, we derive
fine-grained bias-variance decompositions for a very general class of random
feature models with structured covariates. These novel results allow us to
discover a scaling regime for random feature models where the variance due to
the features limits performance in the overparameterized setting. We also
demonstrate how anisotropic weight structure in random feature models can limit
performance and lead to nontrivial exponents for finite-width corrections in
the overparameterized setting. Our results extend and provide a unifying
perspective on earlier models of neural scaling laws.

用随机矩阵理论和自由概率的基本工具简要推导了多种高维岭回归模型的训练和泛化性能，在物理学和深度学习背景的读者中提供了这些主题的介绍和评论。通过自由概率的 $S$ 变换特性，从代数的几行直接获得训练和泛化误差的解析公式，能够直观地识别模型性能的幂律缩放来源。计算了广义类随机特征模型的泛化误差，发现在所有模型中，$S$ 变换对应于训练 - 测试泛化差距，并提供了广义交叉验证估计器的类比。利用这些技术，对具有结构化协变量的非常通用的随机特征模型得到了细粒度的偏差 - 方差分解。这些新颖结果使我们能够发现随机特征模型的缩放区域，在超参数设置中特征的方差限制了性能。我们还演示了随机特征模型中异向权重结构如何限制性能，并导致超参数设置中有限宽度修正的非平凡指数。我们的结果扩展并提供了对早期神经缩放定律模型的统一视角。

高维回归中的缩放和重标定

Scaling and renormalization in high-dimensional regression

We revisit the initialization of deep residual networks (ResNets) by
introducing a novel analytical tool in free probability to the community of
deep learning. This tool deals with non-Hermitian random matrices, rather than
their conventional Hermitian counterparts in the literature. As a consequence,
this new tool enables us to evaluate the singular value spectrum of the
input-output Jacobian of a fully-connected deep ResNet for both linear and
nonlinear cases. With the powerful tool of free probability, we conduct an
asymptotic analysis of the spectrum on the single-layer case, and then extend
this analysis to the multi-layer case of an arbitrary number of layers. In
particular, we propose to rescale the classical random initialization by the
number of residual units, so that the spectrum has the order of $O(1)$, when
compared with the large width and depth of the network. We empirically
demonstrate that the proposed initialization scheme learns at a speed of orders
of magnitudes faster than the classical ones, and thus attests a strong
practical relevance of this investigation.

使用自由概率的强大工具对深度残差网络（ResNets）的初始化进行重审，并建议按残差单元数量重新缩放经典随机初始化，以获得数量级更高的学习速度。

深度残差学习中的频谱集中：一种自由概率方法

Spectrum concentration in deep residual learning: a free probability  approach

The concept of freeness was introduced by Voiculescu in the context of
operator algebras. Later it was observed that it is also relevant for large
random matrices. We will show how the combination of various free probability
results with a linearization trick allows to address successfully the problem
of determining the asymptotic eigenvalue distribution of general selfadjoint
polynomials in independent random matrices.

本文介绍了 Voiculescu 提出的自由性概念，讨论了其在算子代数和大型随机矩阵中的应用，通过线性化技巧结合各种自由概率结果成功地解决了确定独立随机矩阵中一般自伴随多项式的渐近特征值分布问题。