Understanding generalization in deep neural networks is an active area of
research. A promising avenue of exploration has been that of margin
measurements: the shortest distance to the decision boundary for a given sample
or that sample's representation internal to the network. Margin-based
complexity measures have been shown to be correlated with the generalization
ability of deep neural networks in some circumstances but not others. The
reasons behind the success or failure of these metrics are currently unclear.
In this study, we examine margin-based generalization prediction methods in
different settings. We motivate why these metrics sometimes fail to accurately
predict generalization and how they can be improved.
First, we analyze the relationship between margins measured in the input
space and sample noise. We find that different types of sample noise can have a
very different effect on the overall margin of a network that has modeled noisy
data.
Following this, we empirically evaluate how robust margins measured at
different representational spaces are at predicting generalization. We find
that these metrics have several limitations and that a large margin does not
exhibit a strong correlation with empirical risk in many cases.
Finally, we introduce a new margin-based measure that incorporates an
approximation of the underlying data manifold. It is empirically demonstrated
that this measure is generally more predictive of generalization than all other
margin-based measures. Furthermore, we find that this measurement also
outperforms other contemporary complexity measures on a well-known
generalization prediction benchmark. In addition, we analyze the utility and
limitations of this approach and find that this metric is well aligned with
intuitions expressed in prior work.

在这项研究中，我们分析了在不同环境下基于边界距离的泛化预测方法，并提出了一种融合基础数据流形的新的基于边界距离的度量，该度量在大多数情况下能更好地预测泛化。同时，我们对这种方法的实用性和局限性进行了分析，并发现这个度量与之前的工作的观点是吻合的。