Few neural architectures lend themselves to provable learning with gradient
based methods. One popular model is the single-index model, in which labels are
produced by composing an unknown linear projection with a possibly unknown
scalar link function. Learning this model with SGD is relatively
well-understood, whereby the so-called information exponent of the link
function governs a polynomial sample complexity rate. However, extending this
analysis to deeper or more complicated architectures remains challenging.
In this work, we consider single index learning in the setting of symmetric
neural networks. Under analytic assumptions on the activation and maximum
degree assumptions on the link function, we prove that gradient flow recovers
the hidden planted direction, represented as a finitely supported vector in the
feature space of power sum polynomials. We characterize a notion of information
exponent adapted to our setting that controls the efficiency of learning.

在对称神经网络的设置下，通过对激活函数进行分析和对连接函数进行最大度数的假设，我们证明了梯度流可以恢复隐藏的预设方向，该方向在幂和多项式特征空间中表示为一个有限支持的向量，并刻画了适应我们设置的信息指数概念来控制学习的效率。

对称单指标学习

Symmetric Single Index Learning

We focus on the task of learning a single index model $\sigma(w^\star \cdot
x)$ with respect to the isotropic Gaussian distribution in $d$ dimensions.
Prior work has shown that the sample complexity of learning $w^\star$ is
governed by the information exponent $k^\star$ of the link function $\sigma$,
which is defined as the index of the first nonzero Hermite coefficient of
$\sigma$. Ben Arous et al. (2021) showed that $n \gtrsim d^{k^\star-1}$ samples
suffice for learning $w^\star$ and that this is tight for online SGD. However,
the CSQ lower bound for gradient based methods only shows that $n \gtrsim
d^{k^\star/2}$ samples are necessary. In this work, we close the gap between
the upper and lower bounds by showing that online SGD on a smoothed loss learns
$w^\star$ with $n \gtrsim d^{k^\star/2}$ samples. We also draw connections to
statistical analyses of tensor PCA and to the implicit regularization effects
of minibatch SGD on empirical losses.

本文研究在高斯分布下学习单指标模型 σ(w⁎・x)，以及学习该模型时信息指数 k * 的影响，论文提出了利用平滑误差函数与在线随机梯度下降等方法，减少样本数对模型的学习。

平滑景观增强了 SGD 的信号：学习单指数模型的最优样本复杂度

Smoothing the Landscape Boosts the Signal for SGD: Optimal Sample  Complexity for Learning Single Index Models

Stochastic gradient descent (SGD) is a popular algorithm for optimization
problems arising in high-dimensional inference tasks. Here one produces an
estimator of an unknown parameter from independent samples of data by
iteratively optimizing a loss function. This loss function is random and often
non-convex. We study the performance of the simplest version of SGD, namely
online SGD, from a random start in the setting where the parameter space is
high-dimensional.
We develop nearly sharp thresholds for the number of samples needed for
consistent estimation as one varies the dimension. Our thresholds depend only
on an intrinsic property of the population loss which we call the information
exponent. In particular, our results do not assume uniform control on the loss
itself, such as convexity or uniform derivative bounds. The thresholds we
obtain are polynomial in the dimension and the precise exponent depends
explicitly on the information exponent. As a consequence of our results, we
find that except for the simplest tasks, almost all of the data is used simply
in the initial search phase to obtain non-trivial correlation with the ground
truth. Upon attaining non-trivial correlation, the descent is rapid and
exhibits law of large numbers type behavior.
We illustrate our approach by applying it to a wide set of inference tasks
such as phase retrieval, and parameter estimation for generalized linear
models, online PCA, and spiked tensor models, as well as to supervised learning
for single-layer networks with general activation functions.

研究了 SGD 算法在高维参数空间下最简单在线版本的性能，通过对样本数量的阈值来确定参数估计的一致性，其阈值是多项式维度的，取决于信息指数。