Understanding the generalization capability of learning algorithms is at the heart of statistical learning theory. In this paper, we investigate the generalization gap of stochastic gradient Langevin dynamics (SGLD), a widely used optimizer for training deep neural networks (DNNs). We derive an algorithm-dependent generalization bound by analyzing SGLD through an information-theoretic lens. Our analysis reveals an intricate trade-off between learning and information dissipation: SGLD learns from data by updating parameters at each iteration while dissipating information from early training stages. Our bound also involves the variance of gradients which captures a particular kind of "sharpness" of the loss landscape. The main proof techniques in this paper rely on strong data processing inequalities -- a fundamental concept in information theory -- and Otto-Villani's HWI inequality. Finally, we demonstrate our bound through numerical experiments, showing that it can predict the behavior of the true generalization gap.

本文研究嘈杂迭代算法对机器学习模型泛化性能的影响，并通过与通信和信息理论中发现的加性噪声通道的联系推导出分布相关的泛化上限，得出了对几种应用的见解，包括差分隐私随机梯度下降（DP-SGD）、联邦学习和随机梯度 Langevin 动力学（SGLD）。我们通过数字实验演示了我们的边界，表明它们可以帮助理解神经网络泛化现象的最新经验观察。

使用加性噪声通道属性的噪声迭代算法的概化界限