Bilevel optimization has found successful applications in various machine
learning problems, including hyper-parameter optimization, data cleaning, and
meta-learning. However, its huge computational cost presents a significant
challenge for its utilization in large-scale problems. This challenge arises
due to the nested structure of the bilevel formulation, where each
hyper-gradient computation necessitates a costly inner optimization procedure.
To address this issue, we propose a reformulation of bilevel optimization as a
minimax problem, effectively decoupling the outer-inner dependency. Under mild
conditions, we show these two problems are equivalent. Furthermore, we
introduce a multi-stage gradient descent and ascent (GDA) algorithm to solve
the resulting minimax problem with convergence guarantees. Extensive
experimental results demonstrate that our method outperforms state-of-the-art
bilevel methods while significantly reducing the computational cost.

提出了一种将双层优化问题重新设计成极小极大问题的方法，并采用渐进式梯度下降上升算法来解决该问题，结果表明该算法在降低计算成本的同时，优于现有的基于双层优化的算法。