Recently, information theoretic analysis has become a popular framework for
understanding the generalization behavior of deep neural networks. It allows a
direct analysis for stochastic gradient/Langevin descent (SGD/SGLD) learning
algorithms without strong assumptions such as Lipschitz or convexity
conditions. However, the current generalization error bounds within this
framework are still far from optimal, while substantial improvements on these
bounds are quite challenging due to the intractability of high-dimensional
information quantities. To address this issue, we first propose a novel
information theoretical measure: kernelized Renyi's entropy, by utilizing
operator representation in Hilbert space. It inherits the properties of
Shannon's entropy and can be effectively calculated via simple random sampling,
while remaining independent of the input dimension. We then establish the
generalization error bounds for SGD/SGLD under kernelized Renyi's entropy,
where the mutual information quantities can be directly calculated, enabling
evaluation of the tightness of each intermediate step. We show that our
information-theoretical bounds depend on the statistics of the stochastic
gradients evaluated along with the iterates, and are rigorously tighter than
the current state-of-the-art (SOTA) results. The theoretical findings are also
supported by large-scale empirical studies1.

该论文提出了一种基于信息理论的方法，使用内核化的 Renyi 熵刻画了随机梯度下降算法的学习效率，实现了更紧密的泛化误差界限。