We present EGN, a stochastic second-order optimization algorithm that
combines the generalized Gauss-Newton (GN) Hessian approximation with low-rank
linear algebra to compute the descent direction. Leveraging the Duncan-Guttman
matrix identity, the parameter update is obtained by factorizing a matrix which
has the size of the mini-batch. This is particularly advantageous for
large-scale machine learning problems where the dimension of the neural network
parameter vector is several orders of magnitude larger than the batch size.
Additionally, we show how improvements such as line search, adaptive
regularization, and momentum can be seamlessly added to EGN to further
accelerate the algorithm. Moreover, under mild assumptions, we prove that our
algorithm converges to an $\epsilon$-stationary point at a linear rate.
Finally, our numerical experiments demonstrate that EGN consistently exceeds,
or at most matches the generalization performance of well-tuned SGD, Adam, and
SGN optimizers across various supervised and reinforcement learning tasks.

我们介绍了 EGN，一种随机二阶优化算法，将广义高斯 - 牛顿（GN）Hessian 近似与低秩线性代数相结合，计算下降方向。借助 Duncan-Guttman 矩阵恒等式，通过分解一个与小批次大小相同的矩阵来获得参数更新，这在维度远超批次大小的大规模机器学习问题中特别有优势。此外，我们展示了如何将线搜索、自适应正则化和动量等改进无缝地添加到 EGN 中以进一步加速算法。此外，在温和假设下，我们证明了我们的算法以线性速率收敛到一个 ε- 稳定点。最后，我们的数值实验表明，在各种监督学习和强化学习任务中，EGN 始终超过或最多与 SGD、Adam 和 SGN 等优化器的泛化性能相匹配。

深度神经网络训练的精确 Gauss-Newton 优化

Exact Gauss-Newton Optimization for Training Deep Neural Networks

The generalized Gauss-Newton (GGN) optimization method incorporates curvature
estimates into its solution steps, and provides a good approximation to the
Newton method for large-scale optimization problems. GGN has been found
particularly interesting for practical training of deep neural networks, not
only for its impressive convergence speed, but also for its close relation with
neural tangent kernel regression, which is central to recent studies that aim
to understand the optimization and generalization properties of neural
networks. This work studies a GGN method for optimizing a two-layer neural
network with explicit regularization. In particular, we consider a class of
generalized self-concordant (GSC) functions that provide smooth approximations
to commonly-used penalty terms in the objective function of the optimization
problem. This approach provides an adaptive learning rate selection technique
that requires little to no tuning for optimal performance. We study the
convergence of the two-layer neural network, considered to be
overparameterized, in the optimization loop of the resulting GGN method for a
given scaling of the network parameters. Our numerical experiments highlight
specific aspects of GSC regularization that help to improve generalization of
the optimized neural network. The code to reproduce the experimental results is
available at this https URL

研究了使用广义高斯 - 牛顿优化方法优化具有显式正则化的双层神经网络，通过考虑常用目标函数中惩罚项的光滑近似来提供自适应学习率选择技术，数值实验结果突出了广义自共轭正则化对优化后的神经网络泛化性能的改善方面。