Knowledge distillation enables fast and effective transfer of features
learned from a bigger model to a smaller one. However, distillation objectives
are susceptible to sub-population shifts, a common scenario in medical imaging
analysis which refers to groups/domains of data that are underrepresented in
the training set. For instance, training models on health data acquired from
multiple scanners or hospitals can yield subpar performance for minority
groups. In this paper, inspired by distributionally robust optimization (DRO)
techniques, we address this shortcoming by proposing a group-aware distillation
loss. During optimization, a set of weights is updated based on the per-group
losses at a given iteration. This way, our method can dynamically focus on
groups that have low performance during training. We empirically validate our
method, GroupDistil on two benchmark datasets (natural images and cardiac MRIs)
and show consistent improvement in terms of worst-group accuracy.

通过组别感知的蒸馏方法，我们通过优化一组权重，动态关注在训练过程中表现不佳的群体，从而解决医学影像分析中的子群体变化问题并改善最差群体的准确性。