Machine learning models trained by different optimization algorithms under
different data distributions can exhibit distinct generalization behaviors. In
this paper, we analyze the generalization of models trained by noisy iterative
algorithms. We derive distribution-dependent generalization bounds by
connecting noisy iterative algorithms to additive noise channels found in
communication and information theory. Our generalization bounds shed light on
several applications, including differentially private stochastic gradient
descent (DP-SGD), federated learning, and stochastic gradient Langevin dynamics
(SGLD). We demonstrate our bounds through numerical experiments, showing that
they can help understand recent empirical observations of the generalization
phenomena of neural networks.

本文研究嘈杂迭代算法对机器学习模型泛化性能的影响，并通过与通信和信息理论中发现的加性噪声通道的联系推导出分布相关的泛化上限，得出了对几种应用的见解，包括差分隐私随机梯度下降（DP-SGD）、联邦学习和随机梯度 Langevin 动力学（SGLD）。我们通过数字实验演示了我们的边界，表明它们可以帮助理解神经网络泛化现象的最新经验观察。