We present a theoretical and empirical study of the gradient dynamics of
overparameterized shallow ReLU networks with one-dimensional input, solving
least-squares interpolation. We show that the gradient dynamics of such
networks are determined by the gradient flow in a non-redundant
parameterization of the network function. We examine the principal qualitative
features of this gradient flow. In particular, we determine conditions for two
learning regimes:kernel and adaptive, which depend both on the relative
magnitude of initialization of weights in different layers and the asymptotic
behavior of initialization coefficients in the limit of large network widths.
We show that learning in the kernel regime yields smooth interpolants,
minimizing curvature, and reduces to cubic splines for uniform initializations.
Learning in the adaptive regime favors instead linear splines, where knots
cluster adaptively at the sample points.

本文介绍了在一个维度输入下，解决最小二乘插值的超参数化浅层 ReLU 网络的梯度动态的理论和实证研究，阐述了这种网络的梯度动态通过网络函数的非冗余参数化来确定，同时还研究了两种学习模式：核函数和自适应函数，以及它们所对应的性质。