In this paper, we propose a structure-guided Gauss-Newton (SgGN) method for
solving least squares problems using a shallow ReLU neural network. The method
effectively takes advantage of both the least squares structure and the neural
network structure of the objective function. By categorizing the weights and
biases of the hidden and output layers of the network as nonlinear and linear
parameters, respectively, the method iterates back and forth between the
nonlinear and linear parameters. The nonlinear parameters are updated by a
damped Gauss-Newton method and the linear ones are updated by a linear solver.
Moreover, at the Gauss-Newton step, a special form of the Gauss-Newton matrix
is derived for the shallow ReLU neural network and is used for efficient
iterations. It is shown that the corresponding mass and Gauss-Newton matrices
in the respective linear and nonlinear steps are symmetric and positive
definite under reasonable assumptions. Thus, the SgGN method naturally produces
an effective search direction without the need of additional techniques like
shifting in the Levenberg-Marquardt method to achieve invertibility of the
Gauss-Newton matrix. The convergence and accuracy of the method are
demonstrated numerically for several challenging function approximation
problems, especially those with discontinuities or sharp transition layers that
pose significant challenges for commonly used training algorithms in machine
learning.

本文提出了一种基于结构引导的 Gauss-Newton 方法，用于使用浅层 ReLU 神经网络求解最小二乘问题。该方法有效地利用了最小二乘结构和目标函数的神经网络结构，通过将网络的隐藏层和输出层的权重和偏置分别归类为非线性和线性参数，方法在非线性和线性参数之间来回迭代。该方法在最小二乘步骤中，为浅层 ReLU 神经网络推导出一种特殊形式的 Gauss-Newton 矩阵，用于高效迭代。实验证明，在合理的假设下，线性和非线性步骤中的对应的质量矩阵和 Gauss-Newton 矩阵均是对称且正定的。因此，SgGN 方法自然地产生了一个有效的搜索方向，无需像 Levenberg-Marquardt 方法中的移位技术一样实现 Gauss-Newton 矩阵的可逆性。该方法的收敛性和准确性在多个具有挑战性的函数逼近问题中进行了数值验证，特别是在机器学习中常用的训练算法在具有不连续性或尖锐过渡层的问题上面临重大挑战的情况下。

一种基于结构引导的浅层 ReLU 神经网络的高斯牛顿方法

A Structure-Guided Gauss-Newton Method for Shallow ReLU Neural Network

We present an efficient block-diagonal ap- proximation to the Gauss-Newton
matrix for feedforward neural networks. Our result- ing algorithm is
competitive against state- of-the-art first order optimisation methods, with
sometimes significant improvement in optimisation performance. Unlike
first-order methods, for which hyperparameter tuning of the optimisation
parameters is often a labo- rious process, our approach can provide good
performance even when used with default set- tings. A side result of our work
is that for piecewise linear transfer functions, the net- work objective
function can have no differ- entiable local maxima, which may partially explain
why such transfer functions facilitate effective optimisation.

我们提出了一种高效的分块对角近似方法来计算前馈神经网络的高斯 - 牛顿矩阵，与一阶优化方法相比，有时能够显著提高优化性能。此外，我们的方法无需繁琐的调参，即可提供良好的性能。在针对分段线性转移函数进行优化时，网络目标函数可能不存在可微的局部极大值，这也可以部分解释为什么这样的转移函数有助于有效优化。