Knowledge distillation~(KD) is an effective learning paradigm for improving
the performance of lightweight student networks by utilizing additional
supervision knowledge distilled from teacher networks. Most pioneering studies
either learn from only a single teacher in their distillation learning methods,
neglecting the potential that a student can learn from multiple teachers
simultaneously, or simply treat each teacher to be equally important, unable to
reveal the different importance of teachers for specific examples. To bridge
this gap, we propose a novel adaptive multi-teacher multi-level knowledge
distillation learning framework~(AMTML-KD), which consists two novel insights:
(i) associating each teacher with a latent representation to adaptively learn
instance-level teacher importance weights which are leveraged for acquiring
integrated soft-targets~(high-level knowledge) and (ii) enabling the
intermediate-level hints~(intermediate-level knowledge) to be gathered from
multiple teachers by the proposed multi-group hint strategy. As such, a student
model can learn multi-level knowledge from multiple teachers through AMTML-KD.
Extensive results on publicly available datasets demonstrate the proposed
learning framework ensures student to achieve improved performance than strong
competitors.

我们提出了基于多教师多层知识蒸馏学习框架的自适应学习方法，该方法通过将每个教师与潜在表示相关联，自适应地学习实例级教师重要性权重，从而获取集成的高级知识，并通过多组提示策略从多个教师处汇集中间级知识。实验表明，该方法确保学生比强竞争者取得了更好的性能。