Traditional knowledge distillation (KD) relies on a proficient teacher
trained on the target task, which is not always available. In this setting,
cross-task distillation can be used, enabling the use of any teacher model
trained on a different task. However, many KD methods prove ineffective when
applied to this cross-task setting. To address this limitation, we propose a
simple modification: the use of an inverted projection. We show that this
drop-in replacement for a standard projector is effective by learning to
disregard any task-specific features which might degrade the student's
performance. We find that this simple modification is sufficient for extending
many KD methods to the cross-task setting, where the teacher and student tasks
can be very different. In doing so, we obtain up to a 1.9% improvement in the
cross-task setting compared to the traditional projection, at no additional
cost. Our method can obtain significant performance improvements (up to 7%)
when using even a randomly-initialised teacher on various tasks such as depth
estimation, image translation, and semantic segmentation, despite the lack of
any learned knowledge to transfer. To provide conceptual and analytical
insights into this result, we show that using an inverted projection allows the
distillation loss to be decomposed into a knowledge transfer and a spectral
regularisation component. Through this analysis we are additionally able to
propose a novel regularisation loss that allows teacher-free distillation,
enabling performance improvements of up to 8.57% on ImageNet with no additional
training costs.

通过使用反向投影的简单修改，我们提出了一种针对跨任务设置的知识蒸馏方法，能够有效处理不同任务之间的教师和学生模型，在没有额外成本的情况下，在跨任务设置下相比传统投影获得了高达 1.9％的改进。此外，我们还使用反向投影进行了分析，提出了一种新的正则化损失函数，使得在 ImageNet 上的无教师蒸馏性能提升高达 8.57％，且无需额外的训练成本。

跨任务知识蒸馏的投影学习

Learning to Project for Cross-Task Knowledge Distillation

Distillation is an effective knowledge-transfer technique that uses predicted
distributions of a powerful teacher model as soft targets to train a
less-parameterized student model. A pre-trained high capacity teacher, however,
is not always available. Recently proposed online variants use the aggregated
intermediate predictions of multiple student models as targets to train each
student model. Although group-derived targets give a good recipe for
teacher-free distillation, group members are homogenized quickly with simple
aggregation functions, leading to early saturated solutions. In this work, we
propose Online Knowledge Distillation with Diverse peers (OKDDip), which
performs two-level distillation during training with multiple auxiliary peers
and one group leader. In the first-level distillation, each auxiliary peer
holds an individual set of aggregation weights generated with an
attention-based mechanism to derive its own targets from predictions of other
auxiliary peers. Learning from distinct target distributions helps to boost
peer diversity for effectiveness of group-based distillation. The second-level
distillation is performed to transfer the knowledge in the ensemble of
auxiliary peers further to the group leader, i.e., the model used for
inference. Experimental results show that the proposed framework consistently
gives better performance than state-of-the-art approaches without sacrificing
training or inference complexity, demonstrating the effectiveness of the
proposed two-level distillation framework.

本文提出了一种双层蒸馏网络框架 OKDDip，其中包括多个辅助对等体和一个领导群体，辅助对等体通过注意力机制获取自己的预测目标，来作为组群内部的目标进行蒸馏学习，然后再将知识传递给领导群体，实现知识转移。实验结果表明，该框架在训练或推理复杂度上不会牺牲性能，相较于现有方法，可以提供更好的蒸馏效果。