We use the PAC-Bayesian theory for the setting of learning-to-optimize. To the best of our knowledge, we present the first framework to learn optimization algorithms with provable generalization guarantees (PAC-Bayesian bounds) and explicit trade-off between convergence guarantees and convergence speed, which contrasts with the typical worst-case analysis. Our learned optimization algorithms provably outperform related ones derived from a (deterministic) worst-case analysis. The results rely on PAC-Bayesian bounds for general, possibly unbounded loss-functions based on exponential families. Then, we reformulate the learning procedure into a one-dimensional minimization problem and study the possibility to find a global minimum. Furthermore, we provide a concrete algorithmic realization of the framework and new methodologies for learning-to-optimize, and we conduct four practically relevant experiments to support our theory. With this, we showcase that the provided learning framework yields optimization algorithms that provably outperform the state-of-the-art by orders of magnitude.

用PAC-Bayesian理论为学习优化问题提供了第一个具备可证估计以及收敛保证和收敛速度权衡的框架，学习出的优化算法在比起仅从最坏情况分析得出的算法上具备可证的优越性能，基于指数族的PAC-Bayesian上界对一般的、可能无界的损失函数提供了可行性，我们通过将学习过程转化为一维最小化问题并研究全局最小值的可能性，提供了一个具体算法实现和学习优化的新方法，并进行了四个实际相关的实验来支持我们的理论，展示出该学习框架使得优化算法的性能有了数个数量级的改进。

具有PAC-Bayesian保证的学习优化：理论考虑和实践实现