This paper studies the gradient flow dynamics that arise when training deep
homogeneous neural networks, starting with small initializations. The present
work considers neural networks that are assumed to have locally Lipschitz
gradients and an order of homogeneity strictly greater than two. This paper
demonstrates that for sufficiently small initializations, during the early
stages of training, the weights of the neural network remain small in norm and
approximately converge in direction along the Karush-Kuhn-Tucker (KKT) points
of the neural correlation function introduced in [1]. Additionally, for square
loss and under a separability assumption on the weights of neural networks, a
similar directional convergence of gradient flow dynamics is shown near certain
saddle points of the loss function.

该论文研究了使用小的初始值训练深层均匀神经网络时产生的梯度流动力学。该研究表明，在训练的早期阶段，神经网络的权重保持较小的范数，并且在神经关联函数的 Karush-Kuhn-Tucker (KKT) 点附近大致收敛于相同方向。此外，在平方损失和神经网络权重的可分离性假设下，梯度流动力学在损失函数的某些鞍点附近也显示出类似的方向收敛。

深度均质神经网络的早期方向收敛及小初始化

Early Directional Convergence in Deep Homogeneous Neural Networks for  Small Initializations

This paper examines gradient flow dynamics of two-homogeneous neural networks
for small initializations, where all weights are initialized near the origin.
For both square and logistic losses, it is shown that for sufficiently small
initializations, the gradient flow dynamics spend sufficient time in the
neighborhood of the origin to allow the weights of the neural network to
approximately converge in direction to the Karush-Kuhn-Tucker (KKT) points of a
neural correlation function that quantifies the correlation between the output
of the neural network and corresponding labels in the training data set. For
square loss, it has been observed that neural networks undergo saddle-to-saddle
dynamics when initialized close to the origin. Motivated by this, this paper
also shows a similar directional convergence among weights of small magnitude
in the neighborhood of certain saddle points.

本文研究了初始接近原点的两均勻神經網絡的梯度流動力學，對於方塊和邏輯損失，會有足夠長的時間在原點的附近，使神經網絡的權重大約收斂於量化神經網絡輸出和相應標籤在訓練數據集上相關性的 Karush-Kuhn-Tucker（KKT）點；方塊損失下神經網絡經常在接近原點的地方產生鞍點，本文在此基礎上展示了小幅度權重在某些鞍點附近的類似方向收斂。