Modern machine learning usually involves predictors in the overparametrised
setting (number of trained parameters greater than dataset size), and their
training yield not only good performances on training data, but also good
generalisation capacity. This phenomenon challenges many theoretical results,
and remains an open problem. To reach a better understanding, we provide novel
generalisation bounds involving gradient terms. To do so, we combine the
PAC-Bayes toolbox with Poincar\'e and Log-Sobolev inequalities, avoiding an
explicit dependency on dimension of the predictor space. Our results highlight
the positive influence of \emph{flat minima} (being minima with a neighbourhood
nearly minimising the learning problem as well) on generalisation performances,
involving directly the benefits of the optimisation phase.

现代机器学习通常涉及超参数设置（训练参数数量大于数据集大小），它们的训练结果不仅在训练数据上表现良好，而且具有良好的泛化能力。本文通过与 PAC-Bayes 工具箱相结合的 Poincaré 和 Log-Sobolev 不等式提供了新的包含梯度项的泛化界限，避免了对预测空间维度的明确依赖。我们的结果突出了 “平坦极小值” 对泛化性能的积极影响，直接涉及了优化阶段的益处。

泛化与平坦极小之间的 PAC-Bayesian 联系

A PAC-Bayesian Link Between Generalisation and Flat Minima

Minimising upper bounds on the population risk or the generalisation gap has
been widely used in structural risk minimisation (SRM) - this is in particular
at the core of PAC-Bayesian learning. Despite its successes and unfailing surge
of interest in recent years, a limitation of the PAC-Bayesian framework is that
most bounds involve a Kullback-Leibler (KL) divergence term (or its
variations), which might exhibit erratic behavior and fail to capture the
underlying geometric structure of the learning problem - hence restricting its
use in practical applications. As a remedy, recent studies have attempted to
replace the KL divergence in the PAC-Bayesian bounds with the Wasserstein
distance. Even though these bounds alleviated the aforementioned issues to a
certain extent, they either hold in expectation, are for bounded losses, or are
nontrivial to minimize in an SRM framework. In this work, we contribute to this
line of research and prove novel Wasserstein distance-based PAC-Bayesian
generalisation bounds for both batch learning with independent and identically
distributed (i.i.d.) data, and online learning with potentially non-i.i.d.
data. Contrary to previous art, our bounds are stronger in the sense that (i)
they hold with high probability, (ii) they apply to unbounded (potentially
heavy-tailed) losses, and (iii) they lead to optimizable training objectives
that can be used in SRM. As a result we derive novel Wasserstein-based
PAC-Bayesian learning algorithms and we illustrate their empirical advantage on
a variety of experiments.

本文介绍了基于 Wasserstein 距离的 PAC-Bayesian 泛化边界，并从分别适用于批量学习与独立同分布数据和在线学习的角度进行了证明，并获得了用于 SRM 的可优化培训目标。