We consider the problem of learning an unknown function $f_{\star}$ on the
$d$-dimensional sphere with respect to the square loss, given i.i.d. samples
$\{(y_i,{\boldsymbol x}_i)\}_{i\le n}$ where ${\boldsymbol x}_i$ is a feature
vector uniformly distributed on the sphere and $y_i=f_{\star}({\boldsymbol
x}_i)+\varepsilon_i$. We study two popular classes of models that can be
regarded as linearizations of two-layers neural networks around a random
initialization: the random features model of Rahimi-Recht (RF); the neural
tangent kernel model of Jacot-Gabriel-Hongler (NT). Both these approaches can
also be regarded as randomized approximations of kernel ridge regression (with
respect to different kernels), and enjoy universal approximation properties
when the number of neurons $N$ diverges, for a fixed dimension $d$.
We consider two specific regimes: the approximation-limited regime, in which
$n=\infty$ while $d$ and $N$ are large but finite; and the sample size-limited
regime in which $N=\infty$ while $d$ and $n$ are large but finite. In the first
regime we prove that if $d^{\ell + \delta} \le N\le d^{\ell+1-\delta}$ for
small $\delta > 0$, then \RF\, effectively fits a degree-$\ell$ polynomial in
the raw features, and \NT\, fits a degree-$(\ell+1)$ polynomial. In the second
regime, both RF and NT reduce to kernel methods with rotationally invariant
kernels. We prove that, if the number of samples is $d^{\ell + \delta} \le n
\le d^{\ell +1-\delta}$, then kernel methods can fit at most a a degree-$\ell$
polynomial in the raw features. This lower bound is achieved by kernel ridge
regression. Optimal prediction error is achieved for vanishing ridge
regularization.

本论文研究了在球面上进行方差损失下的未知函数 f * 的学习问题，并研究了神经切向核模型和 Rahimi-Recht 的随机特征模型等两种流行的模型，以及核岭回归。同时，论文探讨了样本数量有限或由于关于度数和样本数的适当估计而未能实现最优化性能时的情况，以及核方法随机选取核函数时的情况。