As deep neural networks (DNNs) achieve tremendous success across many
application domains, researchers tried to explore in many aspects on why they
generalize well. In this paper, we provide a novel perspective on these issues
using the gradient signal to noise ratio (GSNR) of parameters during training
process of DNNs. The GSNR of a parameter is defined as the ratio between its
gradient's squared mean and variance, over the data distribution. Based on
several approximations, we establish a quantitative relationship between model
parameters' GSNR and the generalization gap. This relationship indicates that
larger GSNR during training process leads to better generalization performance.
Moreover, we show that, different from that of shallow models (e.g. logistic
regression, support vector machines), the gradient descent optimization
dynamics of DNNs naturally produces large GSNR during training, which is
probably the key to DNNs' remarkable generalization ability.

本文利用深度神经网络的斜率噪声比率（GSNR）提出了新的模型参数研究视角，建立了模型参数的 GSNR 和泛化差距之间的定量关系，并展示了相对于浅层模型，DNNs 的梯度下降优化动态自然产生大 GSNR，这可能是 DNNs 出色的泛化能力的关键。