Knowledge distillation (KD) is essentially a process of transferring a
teacher model's behavior, e.g., network response, to a student model. The
network response serves as additional supervision to formulate the machine
domain, which uses the data collected from the human domain as a transfer set.
Traditional KD methods hold an underlying assumption that the data collected in
both human domain and machine domain are both independent and identically
distributed (IID). We point out that this naive assumption is unrealistic and
there is indeed a transfer gap between the two domains. Although the gap offers
the student model external knowledge from the machine domain, the imbalanced
teacher knowledge would make us incorrectly estimate how much to transfer from
teacher to student per sample on the non-IID transfer set. To tackle this
challenge, we propose Inverse Probability Weighting Distillation (IPWD) that
estimates the propensity score of a training sample belonging to the machine
domain, and assigns its inverse amount to compensate for under-represented
samples. Experiments on CIFAR-100 and ImageNet demonstrate the effectiveness of
IPWD for both two-stage distillation and one-stage self-distillation.

该研究论文提出了一种新的逆概率加权蒸馏 (IPWD) 方法，用于解决知识蒸馏时两个领域之间数据分布不同的问题，该方法通过对样本的倾向得分估计进行加权，以弥补非独立同分布的数据中低频样本的不足，有效提高了知识蒸馏的准确度。