It is well known that the class of rotation invariant algorithms are
suboptimal even for learning sparse linear problems when the number of examples
is below the "dimension" of the problem. This class includes any gradient
descent trained neural net with a fully-connected input layer (initialized with
a rotationally symmetric distribution). The simplest sparse problem is learning
a single feature out of $d$ features. In that case the classification error or
regression loss grows with $1-k/n$ where $k$ is the number of examples seen.
These lower bounds become vacuous when the number of examples $k$ reaches the
dimension $d$.
We show that when noise is added to this sparse linear problem, rotation
invariant algorithms are still suboptimal after seeing $d$ or more examples. We
prove this via a lower bound for the Bayes optimal algorithm on a rotationally
symmetrized problem. We then prove much lower upper bounds on the same problem
for simple non-rotation invariant algorithms. Finally we analyze the gradient
flow trajectories of many standard optimization algorithms in some simple cases
and show how they veer toward or away from the sparse targets.
We believe that our trajectory categorization will be useful in designing
algorithms that can exploit sparse targets and our method for proving lower
bounds will be crucial for analyzing other families of algorithms that admit
different classes of invariances.

添加噪音后，旋转不变算法在观察到 d 个或更多实例之后仍然是次优的；我们通过对旋转对称问题的贝叶斯最优算法的一个下界证明了这一点，并对简单的非旋转不变算法在同一问题中进行了更低的上界证明；最后，我们分析了一些简单情况下许多标准优化算法的梯度流轨迹，展示了它们朝着或远离稀疏目标的方向发展；我们相信我们的轨迹分类方法在设计能够利用稀疏目标的算法中将是有用的，而我们证明下界的方法对于分析其他拥有不同不变性类别的算法族将是至关重要的。

噪声误导稀疏目标上的旋转不变算法

Noise misleads rotation invariant algorithms on sparse targets

We provide a detailed asymptotic study of gradient flow trajectories and
their implicit optimization bias when minimizing the exponential loss over
"diagonal linear networks". This is the simplest model displaying a transition
between "kernel" and non-kernel ("rich" or "active") regimes. We show how the
transition is controlled by the relationship between the initialization scale
and how accurately we minimize the training loss. Our results indicate that
some limit behaviors of gradient descent only kick in at ridiculous training
accuracies (well beyond $10^{-100}$). Moreover, the implicit bias at reasonable
initialization scales and training accuracies is more complex and not captured
by these limits.

研究了在 “对角线线性网络” 上最小化指数损失的梯度流轨迹及其隐式优化偏差的详细渐近研究，揭示了 “核” 与非 “核”（“丰富” 或 “活跃”）状态之间的转换如何受初始缩放和最小化训练损失的精度之间的关系控制，结果表明，梯度下降的某些极限行为仅在荒谬的训练精度（远远低于 $10^{-100}$）时才能发挥作用。此外，在合理的初始化尺度和训练精度下，隐式偏差更为复杂，超出了这些极限的范畴。