Knowledge distillation is an effective paradigm for boosting the performance
of pocket-size model, especially when multiple teacher models are available,
the student would break the upper limit again. However, it is not economical to
train diverse teacher models for the disposable distillation. In this paper, we
introduce a new concept dubbed Avatars for distillation, which are the
inference ensemble models derived from the teacher. Concretely, (1) For each
iteration of distillation training, various Avatars are generated by a
perturbation transformation. We validate that Avatars own higher upper limit of
working capacity and teaching ability, aiding the student model in learning
diverse and receptive knowledge perspectives from the teacher model. (2) During
the distillation, we propose an uncertainty-aware factor from the variance of
statistical differences between the vanilla teacher and Avatars, to adjust
Avatars' contribution on knowledge transfer adaptively. Avatar Knowledge
Distillation AKD is fundamentally different from existing methods and refines
with the innovative view of unequal training. Comprehensive experiments
demonstrate the effectiveness of our Avatars mechanism, which polishes up the
state-of-the-art distillation methods for dense prediction without more extra
computational cost. The AKD brings at most 0.7 AP gains on COCO 2017 for Object
Detection and 1.83 mIoU gains on Cityscapes for Semantic Segmentation,
respectively.

本文提出一种新的知识蒸馏方法 ——Avatar Knowledge Distillation (AKD)，通过生成因扰动转化而得的不同推理集合模型（Avatars）来协助学生模型更有效的学习，同时提出了一种基于推理模型差异的不确定性感知因素，以适应地调整 Avatars 的贡献，这种方法在不增加额外计算成本的前提下，将目标检测的平均精度提高了 0.7 AP，并将语义分割的平均 IoU 提高了 1.83。