Neural scaling laws aim to characterize how out-of-sample error behaves as a
function of model and training dataset size. Such scaling laws guide allocation
of a computational resources between model and data processing to minimize
error. However, existing theoretical support for neural scaling laws lacks
rigor and clarity, entangling the roles of information and optimization. In
this work, we develop rigorous information-theoretic foundations for neural
scaling laws. This allows us to characterize scaling laws for data generated by
a two-layer neural network of infinite width. We observe that the optimal
relation between data and model size is linear, up to logarithmic factors,
corroborating large-scale empirical investigations. Concise yet general results
of the kind we establish may bring clarity to this topic and inform future
investigations.

通过建立信息论基础，我们研究了神经缩放定律，并揭示了在无限宽度的两层神经网络生成的数据中，数据和模型大小的最佳关系大致呈线性，进一步验证了大规模的实证研究结果。

神经网络规模定律的信息论基础

Information-Theoretic Foundations for Neural Scaling Laws

In the absence of explicit regularization, Kernel "Ridgeless" Regression with
nonlinear kernels has the potential to fit the training data perfectly. It has
been observed empirically, however, that such interpolated solutions can still
generalize well on test data. We isolate a phenomenon of implicit
regularization for minimum-norm interpolated solutions which is due to a
combination of high dimensionality of the input data, curvature of the kernel
function, and favorable geometric properties of the data such as an eigenvalue
decay of the empirical covariance and kernel matrices. In addition to deriving
a data-dependent upper bound on the out-of-sample error, we present
experimental evidence suggesting that the phenomenon occurs in the MNIST
dataset.

使用不带显式正则化的核 “无岭” 回归及非线性核函数能完美拟合训练数据，本文分离了最小范数插值解的隐含正则化现象，这是由于输入数据的高维性、核函数的曲率以及数据的几何特性所导致的，并给出了一种数据相关的外样本误差的上界估计。