We provide novel information-theoretic generalization bounds for stochastic gradient Langevin dynamics (SGLD) under the assumptions of smoothness and dissipativity, which are widely used in sampling and non-convex optimization studies. Our bounds are time-independent and decay to zero as the sample size increases, regardless of the number of iterations and whether the step size is fixed. Unlike previous studies, we derive the generalization error bounds by focusing on the time evolution of the Kullback--Leibler divergence, which is related to the stability of datasets and is the upper bound of the mutual information between output parameters and an input dataset. Additionally, we establish the first information-theoretic generalization bound when the training and test loss are the same by showing that a loss function of SGLD is sub-exponential. This bound is also time-independent and removes the problematic step size dependence in existing work, leading to an improved excess risk bound by combining our analysis with the existing non-convex optimization error bounds.

我们提供了一种新的信息理论泛化界限，用于研究随机梯度Langevin动力学（SGLD），在平滑性和耗散性的假设下。我们的界限是独立于时间的，当样本大小增加时会衰减为零，无论迭代次数和步长是否固定。与以前的研究不同，我们通过关注Kullback-Leibler散度的时间演化来推导泛化错误界限，这与数据集的稳定性有关，并且是输出参数和输入数据集之间互信息的上界。此外，我们通过展示SGLD的损失函数是次指数的来建立了第一个信息理论的泛化界限。这种界限也是独立于时间的，并消除了现有工作中的步长依赖问题，通过将我们的分析与现有的非凸优化误差界限相结合，导致改进的超额风险界限。

SGLD的独立于时间的信息论泛化界