Bilevel optimization (BO) is useful for solving a variety of important
machine learning problems including but not limited to hyperparameter
optimization, meta-learning, continual learning, and reinforcement learning.
Conventional BO methods need to differentiate through the low-level
optimization process with implicit differentiation, which requires expensive
calculations related to the Hessian matrix. There has been a recent quest for
first-order methods for BO, but the methods proposed to date tend to be
complicated and impractical for large-scale deep learning applications. In this
work, we propose a simple first-order BO algorithm that depends only on
first-order gradient information, requires no implicit differentiation, and is
practical and efficient for large-scale non-convex functions in deep learning.
We provide non-asymptotic convergence analysis of the proposed method to
stationary points for non-convex objectives and present empirical results that
show its superior practical performance.

本文提出了一种基于一阶梯度信息的简单双层优化算法，适用于深度学习中大规模的非凸函数，无需隐式微分，并有指导其在非凸优化问题上收敛于驻点的收敛性分析证明，实验结果表明其优越的性能表现。