Knowledge distillation (KD) has shown great potential for transferring
knowledge from a complex teacher model to a simple student model in which the
heavy learning task can be accomplished efficiently and without losing too much
prediction accuracy. Recently, many attempts have been made by applying the KD
mechanism to the graph representation learning models such as graph neural
networks (GNNs) to accelerate the model's inference speed via student models.
However, many existing KD-based GNNs utilize MLP as a universal approximator in
the student model to imitate the teacher model's process without considering
the graph knowledge from the teacher model. In this work, we provide a KD-based
framework on multi-scaled GNNs, known as graph framelet, and prove that by
adequately utilizing the graph knowledge in a multi-scaled manner provided by
graph framelet decomposition, the student model is capable of adapting both
homophilic and heterophilic graphs and has the potential of alleviating the
over-squashing issue with a simple yet effectively graph surgery. Furthermore,
we show how the graph knowledge supplied by the teacher is learned and digested
by the student model via both algebra and geometry. Comprehensive experiments
show that our proposed model can generate learning accuracy identical to or
even surpass the teacher model while maintaining the high speed of inference.

本文提供了一种基于多尺度图框架的知识蒸馏方法，通过合理利用图框架分解所提供的多尺度图知识，学生模型能够适应同构和异构图，并具有通过简单而有效的图手术缓解过度压缩问题的潜力。实验结果表明，该模型可以在保持高推理速度的同时，生成与教师模型相同甚至更好的学习精度。