We present a general approach, based on exponential inequalities, to derive
bounds on the generalization error of randomized learning algorithms. Using
this approach, we provide bounds on the average generalization error as well as
bounds on its tail probability, for both the PAC-Bayesian and single-draw
scenarios. Specifically, for the case of sub-Gaussian loss functions, we obtain
novel bounds that depend on the information density between the training data
and the output hypothesis. When suitably weakened, these bounds recover many of
the information-theoretic bounds available in the literature. We also extend
the proposed exponential-inequality approach to the setting recently introduced
by Steinke and Zakynthinou (2020), where the learning algorithm depends on a
randomly selected subset of the available training data. For this setup, we
present bounds for bounded loss functions in terms of the conditional
information density between the output hypothesis and the random variable
determining the subset choice, given all training data. Through our approach,
we recover the average generalization bound presented by Steinke and
Zakynthinou (2020) and extend it to the PAC-Bayesian and single-draw scenarios.
For the single-draw scenario, we also obtain novel bounds in terms of the
conditional $\alpha$-mutual information and the conditional maximal leakage.

通过指数不等式的方法，我们研究了随机学习算法的泛化误差的界限和概率分布，针对亚高斯损失函数提供了以训练数据和输出假设之间信息密度为依据的新的界限，并将该方法扩展到了基于随机选择训练数据子集的情况。

信息密度和条件信息密度的泛化界

Generalization Bounds via Information Density and Conditional  Information Density

We study the generalization error of randomized learning algorithms --
focusing on stochastic gradient descent (SGD) -- using a novel combination of
PAC-Bayes and algorithmic stability. Importantly, our generalization bounds
hold for all posterior distributions on an algorithm's random hyperparameters,
including distributions that depend on the training data. This inspires an
adaptive sampling algorithm for SGD that optimizes the posterior at runtime. We
analyze this algorithm in the context of our generalization bounds and evaluate
it on a benchmark dataset. Our experiments demonstrate that adaptive sampling
can reduce empirical risk faster than uniform sampling while also improving
out-of-sample accuracy.

通过 PAC-Bayes 和算法稳定性的结合研究了随机梯度下降算法的泛化误差，提出了一种基于后验优化的自适应采样算法，并在基准数据集上进行评估。结果表明，相较于均匀采样，自适应采样既可以更快地降低经验风险，也可以提高样本外准确性。