First-order methods, such as gradient descent (GD) and stochastic gradient descent (SGD), have been proven effective in training neural networks. In the context of over-parameterization, there is a line of work demonstrating that randomly initialized (stochastic) gradient descent converges to a globally optimal solution at a linear convergence rate for the quadratic loss function. However, the learning rate of GD for training two-layer neural networks exhibits poor dependence on the sample size and the Gram matrix, leading to a slow training process. In this paper, we show that for the $L^2$ regression problems, the learning rate can be improved from $\mathcal{O}(\lambda_0/n^2)$ to $\mathcal{O}(1/\|\bm{H}^{\infty}\|_2)$, which implies that GD actually enjoys a faster convergence rate. Furthermore, we generalize the method to GD in training two-layer Physics-Informed Neural Networks (PINNs), showing a similar improvement for the learning rate. Although the improved learning rate has a mild dependence on the Gram matrix, we still need to set it small enough in practice due to the unknown eigenvalues of the Gram matrix. More importantly, the convergence rate is tied to the least eigenvalue of the Gram matrix, which can lead to slow convergence. In this work, we provide the convergence analysis of natural gradient descent (NGD) in training two-layer PINNs, demonstrating that the learning rate can be $\mathcal{O}(1)$, and at this rate, the convergence rate is independent of the Gram matrix.

本研究解决了在过参数化设置下，传统的梯度下降法在样本大小和Gram矩阵依赖性差导致训练缓慢的问题。我们提出了一种新的改进学习率方法，使其在L2回归问题中从$\mathcal{O}(\lambda_0/n^2)$提升到$\mathcal{O}(1/\|\bm{H}^{\infty}\|_2)$，并将其扩展至物理信息神经网络，展示了更快的收敛速度。这一发现将提高PINNs的训练效率，具有重要的应用潜力。

超参数自然梯度下降在物理信息神经网络中的收敛性分析