A novel first-order method is proposed for training generative adversarial
networks (GANs). It modifies the Gauss-Newton method to approximate the min-max
Hessian and uses the Sherman-Morrison inversion formula to calculate the
inverse. The method corresponds to a fixed-point method that ensures necessary
contraction. To evaluate its effectiveness, numerical experiments are conducted
on various datasets commonly used in image generation tasks, such as MNIST,
Fashion MNIST, CIFAR10, FFHQ, and LSUN. Our method is capable of generating
high-fidelity images with greater diversity across multiple datasets. It also
achieves the highest inception score for CIFAR10 among all compared methods,
including state-of-the-art second-order methods. Additionally, its execution
time is comparable to that of first-order min-max methods.

提出了一种新的一阶方法用于训练生成对抗网络（GANs），它修改了高斯 - 牛顿法来近似求解最小 - 最大海森矩阵，并使用谢尔曼 - 莫里森逆公式计算其逆，通过固定点方法确保必要的收敛。通过在 MNIST、Fashion MNIST、CIFAR10、FFHQ 和 LSUN 等常用的图像生成任务的数据集上进行数值实验证明了该方法的有效性。该方法能够在多个数据集上生成高保真度图像，并在与包括最先进的二阶方法在内的其他方法比较中，在 CIFAR10 上获得了最高的 Inception Score，执行时间与一阶最小 - 最大方法相当。

生成对抗网络中极小 - 极大优化问题的高斯 - 牛顿方法

A Gauss-Newton Approach for Min-Max Optimization in Generative  Adversarial Networks

Bilevel optimization has various applications such as hyper-parameter
optimization and meta-learning. Designing theoretically efficient algorithms
for bilevel optimization is more challenging than standard optimization because
the lower-level problem defines the feasibility set implicitly via another
optimization problem. One tractable case is when the lower-level problem
permits strong convexity. Recent works show that second-order methods can
provably converge to an $\epsilon$-first-order stationary point of the problem
at a rate of $\tilde{\mathcal{O}}(\epsilon^{-2})$, yet these algorithms require
a Hessian-vector product oracle. Kwon et al. (2023) resolved the problem by
proposing a first-order method that can achieve the same goal at a slower rate
of $\tilde{\mathcal{O}}(\epsilon^{-3})$. In this work, we provide an improved
analysis demonstrating that the first-order method can also find an
$\epsilon$-first-order stationary point within $\tilde
{\mathcal{O}}(\epsilon^{-2})$ oracle complexity, which matches the upper bounds
for second-order methods in the dependency on $\epsilon$. Our analysis further
leads to simple first-order algorithms that can achieve similar near-optimal
rates in finding second-order stationary points and in distributed bilevel
problems.

本文讨论了双层优化问题，提出了第一和第二阶段方法，探究了优化的复杂度和速度，并提出了适用于分布式双层问题的简单算法。

双层优化中寻找静止点的近最优全一阶算法

Near-Optimal Fully First-Order Algorithms for Finding Stationary Points  in Bilevel Optimization

Recently several methods were proposed for sparse optimization which make
careful use of second-order information [10, 28, 16, 3] to improve local
convergence rates. These methods construct a composite quadratic approximation
using Hessian information, optimize this approximation using a first-order
method, such as coordinate descent and employ a line search to ensure
sufficient descent. Here we propose a general framework, which includes
slightly modified versions of existing algorithms and also a new algorithm,
which uses limited memory BFGS Hessian approximations, and provide a novel
global convergence rate analysis, which covers methods that solve subproblems
via coordinate descent.

提出了一种稀疏优化的通用框架，使用了二阶信息和有限的 BFGS Hessian 逼近，通过坐标下降的方法解决问题，并对其全局收敛率进行了新颖的分析。