Recently, the teacher-student knowledge distillation framework has
demonstrated its potential in training Graph Neural Networks (GNNs). However,
due to the difficulty of training over-parameterized GNN models, one may not
easily obtain a satisfactory teacher model for distillation. Furthermore, the
inefficient training process of teacher-student knowledge distillation also
impedes its applications in GNN models. In this paper, we propose the first
teacher-free knowledge distillation method for GNNs, termed GNN
Self-Distillation (GNN-SD), that serves as a drop-in replacement of the
standard training process. The method is built upon the proposed neighborhood
discrepancy rate (NDR), which quantifies the non-smoothness of the embedded
graph in an efficient way. Based on this metric, we propose the adaptive
discrepancy retaining (ADR) regularizer to empower the transferability of
knowledge that maintains high neighborhood discrepancy across GNN layers. We
also summarize a generic GNN-SD framework that could be exploited to induce
other distillation strategies. Experiments further prove the effectiveness and
generalization of our approach, as it brings: 1) state-of-the-art GNN
distillation performance with less training cost, 2) consistent and
considerable performance enhancement for various popular backbones.

本文提出了基于邻近差异率（NDR）的自适应差异保留（ADR）正则化器以加强知识的传递，进而提出了一种基于 GNN-SD 框架的通用图神经网络知识蒸馏方法，它在减少训练成本的同时实现了领先的蒸馏效果，并为多种流行的骨干网路提供了一致有效的性能增强。