This work focuses on addressing two major challenges in the context of
large-scale nonconvex Bi-Level Optimization (BLO) problems, which are
increasingly applied in machine learning due to their ability to model nested
structures. These challenges involve ensuring computational efficiency and
providing theoretical guarantees. While recent advances in scalable BLO
algorithms have primarily relied on lower-level convexity simplification, our
work specifically tackles large-scale BLO problems involving nonconvexity in
both the upper and lower levels. We simultaneously address computational and
theoretical challenges by introducing an innovative single-loop gradient-based
algorithm, utilizing the Moreau envelope-based reformulation, and providing
non-asymptotic convergence analysis for general nonconvex BLO problems.
Notably, our algorithm relies solely on first-order gradient information,
enhancing its practicality and efficiency, especially for large-scale BLO
learning tasks. We validate our approach's effectiveness through experiments on
various synthetic problems, two typical hyper-parameter learning tasks, and a
real-world neural architecture search application, collectively demonstrating
its superior performance.

该研究聚焦于解决大规模非凸双层优化问题中的两个主要挑战，即确保计算效率和提供理论保证，并通过引入一种创新的基于梯度的单循环算法、利用 Moreau 包络重构以及针对一般非凸双层优化问题提供的非渐进收敛分析，同时解决计算和理论挑战。该算法仅依赖一阶梯度信息，增强了其实用性和效率，特别适用于大规模双层优化学习任务，并通过在各种合成问题、两个典型超参数学习任务和一个真实的神经架构搜索应用上的实验证实了其卓越性能。