The recent paper by Byrd & Lipton (2019), based on empirical observations,
raises a major concern on the impact of importance weighting for the
over-parameterized deep learning models. They observe that as long as the model
can separate the training data, the impact of importance weighting diminishes
as the training proceeds. Nevertheless, there lacks a rigorous characterization
of this phenomenon. In this paper, we provide formal characterizations and
theoretical justifications on the role of importance weighting with respect to
the implicit bias of gradient descent and margin-based learning theory. We
reveal both the optimization dynamics and generalization performance under deep
learning models. Our work not only explains the various novel phenomenons
observed for importance weighting in deep learning, but also extends to the
studies where the weights are being optimized as part of the model, which
applies to a number of topics under active research.

通过对深度学习模型中重要性加权的研究，本文提供了正式的表征和理论证明，并揭示了在深度学习模型下的优化动态和泛化性能，扩展到了许多活跃研究领域。