This research proposes to use the Moreau-Yosida envelope to stabilize the
convergence behavior of bi-level Hyperparameter optimization solvers, and
introduces the new algorithm called Moreau-Yosida regularized Hyperparameter
Optimization (MY-HPO) algorithm. Theoretical analysis on the correctness of the
MY-HPO solution and initial convergence analysis is also provided. Our
empirical results show significant improvement in loss values for a fixed
computation budget, compared to the state-of-art bi-level HPO solvers.

本研究提出了使用 Moreau-Yosida 包络来稳定双层超参数优化求解器的收敛行为，并引入了名为 Moreau-Yosida 正则化超参数优化（MY-HPO）算法的新算法。该文还对 MY-HPO 解的正确性和初始收敛性分析进行了理论分析。实证结果表明与最先进的双层 HPO 求解器相比，对于固定的计算预算，我们的算法在损失值上实现了显着改进。

利用 Moreau-Yosida 正则化稳定双层超参数优化

Stabilizing Bi-Level Hyperparameter Optimization using Moreau-Yosida  Regularization

This paper presents a detailed theoretical analysis of the Langevin Monte
Carlo sampling algorithm recently introduced in Durmus et al. (Efficient
Bayesian computation by proximal Markov chain Monte Carlo: when Langevin meets
Moreau, 2016) when applied to log-concave probability distributions that are
restricted to a convex body $\mathsf{K}$. This method relies on a
regularisation procedure involving the Moreau-Yosida envelope of the indicator
function associated with $\mathsf{K}$. Explicit convergence bounds in total
variation norm and in Wasserstein distance of order $1$ are established. In
particular, we show that the complexity of this algorithm given a first order
oracle is polynomial in the dimension of the state space. Finally, some
numerical experiments are presented to compare our method with competing MCMC
approaches from the literature.

本文对应用于限制在凸体上的对数凸概率分布的 Langevin Monte Carlo 采样算法进行了详细的理论分析，该方法依赖于涉及与 K 相关的指示函数的 Moreau-Yosida 包络的正则化过程，建立了总变差范数和一阶 Wasserstein 距离的显式收敛界限，并且给出了有限状态空间维数的算法复杂度是多项式级别的证明。最后，我们提供了一些数值实验，与文献中的竞争 MCMC 方法进行比较。