The ability of learning useful features is one of the major advantages of
neural networks. Although recent works show that neural network can operate in
a neural tangent kernel (NTK) regime that does not allow feature learning, many
works also demonstrate the potential for neural networks to go beyond NTK
regime and perform feature learning. Recently, a line of work highlighted the
feature learning capabilities of the early stages of gradient-based training.
In this paper we consider another mechanism for feature learning via gradient
descent through a local convergence analysis. We show that once the loss is
below a certain threshold, gradient descent with a carefully regularized
objective will capture ground-truth directions. Our results demonstrate that
feature learning not only happens at the initial gradient steps, but can also
occur towards the end of training.

通过本地收敛分析，该论文展示了梯度下降通过精心正则化的目标函数在损失降至一定阈值以下后能够捕捉到真实方向，从而证明了特征学习不仅发生在初始梯度步骤，也可能发生在训练结束时。