It is well known that the class of rotation invariant algorithms are
suboptimal even for learning sparse linear problems when the number of examples
is below the "dimension" of the problem. This class includes any gradient
descent trained neural net with a fully-connected input layer (initialized with
a rotationally symmetric distribution). The simplest sparse problem is learning
a single feature out of $d$ features. In that case the classification error or
regression loss grows with $1-k/n$ where $k$ is the number of examples seen.
These lower bounds become vacuous when the number of examples $k$ reaches the
dimension $d$.
We show that when noise is added to this sparse linear problem, rotation
invariant algorithms are still suboptimal after seeing $d$ or more examples. We
prove this via a lower bound for the Bayes optimal algorithm on a rotationally
symmetrized problem. We then prove much lower upper bounds on the same problem
for simple non-rotation invariant algorithms. Finally we analyze the gradient
flow trajectories of many standard optimization algorithms in some simple cases
and show how they veer toward or away from the sparse targets.
We believe that our trajectory categorization will be useful in designing
algorithms that can exploit sparse targets and our method for proving lower
bounds will be crucial for analyzing other families of algorithms that admit
different classes of invariances.

添加噪音后，旋转不变算法在观察到 d 个或更多实例之后仍然是次优的；我们通过对旋转对称问题的贝叶斯最优算法的一个下界证明了这一点，并对简单的非旋转不变算法在同一问题中进行了更低的上界证明；最后，我们分析了一些简单情况下许多标准优化算法的梯度流轨迹，展示了它们朝着或远离稀疏目标的方向发展；我们相信我们的轨迹分类方法在设计能够利用稀疏目标的算法中将是有用的，而我们证明下界的方法对于分析其他拥有不同不变性类别的算法族将是至关重要的。