We introduce a data-driven approach to analyze the performance of continuous
optimization algorithms using generalization guarantees from statistical
learning theory. We study classical and learned optimizers to solve families of
parametric optimization problems. We build generalization guarantees for
classical optimizers, using a sample convergence bound, and for learned
optimizers, using the Probably Approximately Correct (PAC)-Bayes framework. To
train learned optimizers, we use a gradient-based algorithm to directly
minimize the PAC-Bayes upper bound. Numerical experiments in signal processing,
control, and meta-learning showcase the ability of our framework to provide
strong generalization guarantees for both classical and learned optimizers
given a fixed budget of iterations. For classical optimizers, our bounds are
much tighter than those that worst-case guarantees provide. For learned
optimizers, our bounds outperform the empirical outcomes observed in their
non-learned counterparts.

介绍了一种数据驱动的方法，利用统计学习理论的泛化保证来分析连续优化算法的性能。研究经典优化器和学习优化器来解决参数优化问题的族群。建立了经典优化器的泛化保证，利用样本收敛界限，以及学习优化器的泛化保证，利用 Probably Approximately Correct (PAC)-Bayes 框架。通过使用基于梯度的算法直接最小化 PAC-Bayes 上界来训练学习优化器。在信号处理、控制和元学习的数值实验中展示了我们的框架能够在固定迭代预算的情况下为经典和学习优化器提供强大的泛化保证。对于经典优化器而言，我们的界限比最坏情况保证的界限要严格得多。对于学习优化器而言，我们的上界优于观察到的非学习对应的实际结果。