Deep networks tend to learn spurious feature-label correlations in real-world
supervised learning tasks. This vulnerability is aggravated in distillation,
where a student model may have lesser representational capacity than the
corresponding teacher model. Often, knowledge of specific spurious correlations
is used to reweight instances & rebalance the learning process. We propose a
novel early readout mechanism whereby we attempt to predict the label using
representations from earlier network layers. We show that these early readouts
automatically identify problem instances or groups in the form of confident,
incorrect predictions. Leveraging these signals to modulate the distillation
loss on an instance level allows us to substantially improve not only group
fairness measures across benchmark datasets, but also overall accuracy of the
student model. We also provide secondary analyses that bring insight into the
role of feature learning in supervision and distillation.

深度网络在现实世界的监督学习任务中往往学习到伪特征 - 标签相关性，而蒸馏中学生模型的表征能力可能比相应的教师模型低，我们提出了一种新颖的早期输出机制，通过使用网络较早层的表征来尝试预测标签，这些早期输出可以自动识别出形式为自信但错误的预测的问题实例或组，利用这些信号在实例级别调节蒸馏损失，不仅能够在基准数据集上大幅改善组公平度度量，还能提高学生模型的整体准确性，此外，我们还进行了次要分析，揭示了特征学习在监督和蒸馏中的作用。