This paper presents a succinct derivation of the training and generalization
performance of a variety of high-dimensional ridge regression models using the
basic tools of random matrix theory and free probability. We provide an
introduction and review of recent results on these topics, aimed at readers
with backgrounds in physics and deep learning. Analytic formulas for the
training and generalization errors are obtained in a few lines of algebra
directly from the properties of the $S$-transform of free probability. This
allows for a straightforward identification of the sources of power-law scaling
in model performance. We compute the generalization error of a broad class of
random feature models. We find that in all models, the $S$-transform
corresponds to the train-test generalization gap, and yields an analogue of the
generalized-cross-validation estimator. Using these techniques, we derive
fine-grained bias-variance decompositions for a very general class of random
feature models with structured covariates. These novel results allow us to
discover a scaling regime for random feature models where the variance due to
the features limits performance in the overparameterized setting. We also
demonstrate how anisotropic weight structure in random feature models can limit
performance and lead to nontrivial exponents for finite-width corrections in
the overparameterized setting. Our results extend and provide a unifying
perspective on earlier models of neural scaling laws.

用随机矩阵理论和自由概率的基本工具简要推导了多种高维岭回归模型的训练和泛化性能，在物理学和深度学习背景的读者中提供了这些主题的介绍和评论。通过自由概率的 $S$ 变换特性，从代数的几行直接获得训练和泛化误差的解析公式，能够直观地识别模型性能的幂律缩放来源。计算了广义类随机特征模型的泛化误差，发现在所有模型中，$S$ 变换对应于训练 - 测试泛化差距，并提供了广义交叉验证估计器的类比。利用这些技术，对具有结构化协变量的非常通用的随机特征模型得到了细粒度的偏差 - 方差分解。这些新颖结果使我们能够发现随机特征模型的缩放区域，在超参数设置中特征的方差限制了性能。我们还演示了随机特征模型中异向权重结构如何限制性能，并导致超参数设置中有限宽度修正的非平凡指数。我们的结果扩展并提供了对早期神经缩放定律模型的统一视角。