Understanding the implicit regularization (or implicit bias) of gradient
descent has recently been a very active research area. However, the implicit
regularization in nonlinear neural networks is still poorly understood,
especially for regression losses such as the square loss. Perhaps surprisingly,
we prove that even for a single ReLU neuron, it is impossible to characterize
the implicit regularization with the square loss by any explicit function of
the model parameters (although on the positive side, we show it can be
characterized approximately). For one hidden-layer networks, we prove a similar
result, where in general it is impossible to characterize implicit
regularization properties in this manner, except for the "balancedness"
property identified in Du et al. [2018]. Our results suggest that a more
general framework than the one considered so far may be needed to understand
implicit regularization for nonlinear predictors, and provides some clues on
what this framework should be.

针对非线性神经网络的回归损失（如平方损失），研究其隐含正则化（即隐含偏差）的特性，虽然已成为一个活跃的研究领域，但迄今为止仍未得到深入探究。本文通过一些证明，表明甚至对于单个 ReLU 神经元，无法利用原模型参数的任何显式函数来表征隐含正则化特性（尽管我们可以近似表征），对于一层隐藏层的网络而言也存在类似的现象。本研究建议采用比目前更加广泛的框架来理解非线性预测的隐性正则化，并提供了一些线索。

ReLU 网络在平方损失下的隐式正则化

Implicit Regularization in ReLU Networks with the Square Loss

Recently, a spate of papers have provided positive theoretical results for
training over-parameterized neural networks (where the network size is larger
than what is needed to achieve low error). The key insight is that with
sufficient over-parameterization, gradient-based methods will implicitly leave
some components of the network relatively unchanged, so the optimization
dynamics will behave as if those components are essentially fixed at their
initial random values. In fact, fixing these explicitly leads to the well-known
approach of learning with random features. In other words, these techniques
imply that we can successfully learn with neural networks, whenever we can
successfully learn with random features. In this paper, we first review these
techniques, providing a simple and self-contained analysis for one-hidden-layer
networks. We then argue that despite the impressive positive results, random
feature approaches are also inherently limited in what they can explain. In
particular, we rigorously show that random features cannot be used to learn
even a single ReLU neuron with standard Gaussian inputs, unless the network
size (or magnitude of the weights) is exponentially large. Since a single
neuron is learnable with gradient-based methods, we conclude that we are still
far from a satisfying general explanation for the empirical success of neural
networks.

本文回顾了最近一系列训练超参数神经网络和学习随机特征的实证结果及其限制性说明，论述了神经网络的理论困境并对其表现出的令人印象深刻的经验结果提出了仍需克服的挑战。