We introduce a data-driven approach to analyze the performance of continuous
optimization algorithms using generalization guarantees from statistical
learning theory. We study classical and learned optimizers to solve families of
parametric optimization problems. We build generalization guarantees for
classical optimizers, using a sample convergence bound, and for learned
optimizers, using the Probably Approximately Correct (PAC)-Bayes framework. To
train learned optimizers, we use a gradient-based algorithm to directly
minimize the PAC-Bayes upper bound. Numerical experiments in signal processing,
control, and meta-learning showcase the ability of our framework to provide
strong generalization guarantees for both classical and learned optimizers
given a fixed budget of iterations. For classical optimizers, our bounds are
much tighter than those that worst-case guarantees provide. For learned
optimizers, our bounds outperform the empirical outcomes observed in their
non-learned counterparts.

介绍了一种数据驱动的方法，利用统计学习理论的泛化保证来分析连续优化算法的性能。研究经典优化器和学习优化器来解决参数优化问题的族群。建立了经典优化器的泛化保证，利用样本收敛界限，以及学习优化器的泛化保证，利用 Probably Approximately Correct (PAC)-Bayes 框架。通过使用基于梯度的算法直接最小化 PAC-Bayes 上界来训练学习优化器。在信号处理、控制和元学习的数值实验中展示了我们的框架能够在固定迭代预算的情况下为经典和学习优化器提供强大的泛化保证。对于经典优化器而言，我们的界限比最坏情况保证的界限要严格得多。对于学习优化器而言，我们的上界优于观察到的非学习对应的实际结果。

基于数据驱动的经典优化器和学习优化器的性能保证

Data-Driven Performance Guarantees for Classical and Learned Optimizers

Generalised Bayesian learning algorithms are increasingly popular in machine
learning, due to their PAC generalisation properties and flexibility. The
present paper aims at providing a self-contained survey on the resulting
PAC-Bayes framework and some of its main theoretical and algorithmic
developments.

本文对 PAC-Bayes 框架及其主要理论和算法发展进行了自包含的调查研究。

PAC-Bayesian 学习初探

A Primer on PAC-Bayesian Learning

Margin-based structured prediction commonly uses a maximum loss over all
possible structured outputs \cite{Altun03,Collins04b,Taskar03}. In natural
language processing, recent work \cite{Zhang14,Zhang15} has proposed the use of
the maximum loss over random structured outputs sampled independently from some
proposal distribution. This method is linear-time in the number of random
structured outputs and trivially parallelizable. We study this family of loss
functions in the PAC-Bayes framework under Gaussian perturbations
\cite{McAllester07}. Under some technical conditions and up to statistical
accuracy, we show that this family of loss functions produces a tighter upper
bound of the Gibbs decoder distortion than commonly used methods. Thus, using
the maximum loss over random structured outputs is a principled way of learning
the parameter of structured prediction models. Besides explaining the
experimental success of \cite{Zhang14,Zhang15}, our theoretical results show
that more general techniques are possible.

在自然语言处理中，最新的工作提出了使用独立采样的随机结构输出的最大损失，该方法在随机结构输出的数量上是线性的，容易并行化。使用这种损失函数最大化损失是参数化的学习规则的合理方式，我们在 PAC-Bayes 框架下的高斯扰动中研究了这种损失函数的家族，并表明它产生比常用方法更紧的上限，因此这是一种更一般的技术。