In weakly supervised learning, unbiased risk estimator(URE) is a powerful tool for training classifiers when training and test data are drawn from different distributions. Nevertheless, UREs lead to overfitting in many problem settings when the models are complex like deep networks. In this paper, we investigate reasons for such overfitting by studying a weakly supervised problem called learning with complementary labels. We argue the quality of gradient estimation matters more in risk minimization. Theoretically, we show that a URE gives an unbiased gradient estimator(UGE). Practically, however, UGEs may suffer from huge variance, which causes empirical gradients to be usually far away from true gradients during minimization. To this end, we propose a novel surrogate complementary loss(SCL) framework that trades zero bias with reduced variance and makes empirical gradients more aligned with true gradients in the direction. Thanks to this characteristic, SCL successfully mitigates the overfitting issue and improves URE-based methods.

该论文研究了使用弱监督学习中的无偏风险估计器训练分类器的问题，解决了由于深度神经网络等模型的复杂性导致URE过度拟合的问题，并提出了一种新的SCL框架来降低方差和改善URE方法，从而实现了用更少的偏差换取更少的方差。

无偏估计可能具有误导性：一项使用互补标签学习的案例研究