Solving a bilevel optimization problem is at the core of several machine
learning problems such as hyperparameter tuning, data denoising, meta- and
few-shot learning, and training-data poisoning. Different from simultaneous or
multi-objective optimization, the steepest descent direction for minimizing the
upper-level cost in a bilevel problem requires the inverse of the Hessian of
the lower-level cost. In this work, we propose a novel algorithm for solving
bilevel optimization problems based on the classical penalty function approach.
Our method avoids computing the Hessian inverse and can handle constrained
bilevel problems easily. We prove the convergence of the method under mild
conditions and show that the exact hypergradient is obtained asymptotically.
Our method's simplicity and small space and time complexities enable us to
effectively solve large-scale bilevel problems involving deep neural networks.
We present results on data denoising, few-shot learning, and training-data
poisoning problems in a large-scale setting. Our results show that our approach
outperforms or is comparable to previously proposed methods based on automatic
differentiation and approximate inversion in terms of accuracy, run-time, and
convergence speed.

本文提出了一种新颖的基于罚函数方法的二层优化问题算法，避免了计算 Hessian 逆矩阵的过程，并可轻松处理限制性二层问题。本方法证明收敛性并在大规模深度神经网络二层问题中表现优异，应用于数据去噪、few-shot 学习和训练数据污染问题，结果表明在准确性、运行时间和收敛速度方面均优于基于自动微分和近似求逆的以前提出的方法。