Sample weighting is widely used in deep learning. A large number of weighting
methods essentially utilize the learning difficulty of training samples to
calculate their weights. In this study, this scheme is called difficulty-based
weighting. Two important issues arise when explaining this scheme. First, a
unified difficulty measure that can be theoretically guaranteed for training
samples does not exist. The learning difficulties of the samples are determined
by multiple factors including noise level, imbalance degree, margin, and
uncertainty. Nevertheless, existing measures only consider a single factor or
in part, but not in their entirety. Second, a comprehensive theoretical
explanation is lacking with respect to demonstrating why difficulty-based
weighting schemes are effective in deep learning. In this study, we
theoretically prove that the generalization error of a sample can be used as a
universal difficulty measure. Furthermore, we provide formal theoretical
justifications on the role of difficulty-based weighting for deep learning,
consequently revealing its positive influences on both the optimization
dynamics and generalization performance of deep models, which is instructive to
existing weighting schemes.

本研究证明了样本的泛化误差可以作为普适的难度度量，进一步提供了关于基于难度加权的有效性的正式理论证据，显示了其对深度模型优化动态和泛化性能的积极影响，具有指导现有加权方案的作用。