Understanding the mechanisms through which neural networks extract statistics from input-label pairs is one of the most important unsolved problems in supervised learning. Prior works have identified that the gram matrices of the weights in trained neural networks of general architectures are proportional to the average gradient outer product of the model, in a statement known as the Neural Feature Ansatz (NFA). However, the reason these quantities become correlated during training is poorly understood. In this work, we explain the emergence of this correlation. We identify that the NFA is equivalent to alignment between the left singular structure of the weight matrices and a significant component of the empirical neural tangent kernels associated with those weights. We establish that the NFA introduced in prior works is driven by a centered NFA that isolates this alignment. We show that the speed of NFA development can be predicted analytically at early training times in terms of simple statistics of the inputs and labels. Finally, we introduce a simple intervention to increase NFA correlation at any given layer, which dramatically improves the quality of features learned.

神经网络从输入-标签对中提取统计数据的机制是监督学习中最重要的未解决问题之一。我们通过解释神经特征假设（NFA）的出现来揭示了这种关联的原因，并且提出了一种简单的干预方法来提高NFA关联性，从而显著改善学习到的特征的质量。

梯度下降法在深度非线性网络中导致权重与经验NTK之间的对齐