Recent years have witnessed the great success of Graph Neural Networks (GNNs)
in handling graph-related tasks. However, MLPs remain the primary workhorse for
practical industrial applications due to their desirable inference efficiency
and scalability. To reduce their gaps, one can directly distill knowledge from
a well-designed teacher GNN to a student MLP, which is termed as GNN-to-MLP
distillation. However, the process of distillation usually entails a loss of
information, and ``which knowledge patterns of GNNs are more likely to be left
and distilled into MLPs?" becomes an important question. In this paper, we
first factorize the knowledge learned by GNNs into low- and high-frequency
components in the spectral domain and then derive their correspondence in the
spatial domain. Furthermore, we identified a potential information drowning
problem for existing GNN-to-MLP distillation, i.e., the high-frequency
knowledge of the pre-trained GNNs may be overwhelmed by the low-frequency
knowledge during distillation; we have described in detail what it represents,
how it arises, what impact it has, and how to deal with it. In this paper, we
propose an efficient Full-Frequency GNN-to-MLP (FF-G2M) distillation framework,
which extracts both low-frequency and high-frequency knowledge from GNNs and
injects it into MLPs. Extensive experiments show that FF-G2M improves over the
vanilla MLPs by 12.6% and outperforms its corresponding teacher GNNs by 2.6%
averaged over six graph datasets and three common GNN architectures.

本文介绍了一种名为 Full-Frequency GNN-to-MLP 的知识蒸馏方法，它能够从 GNN 中抽取低频和高频知识，并将其注入 MLP 中，从而解决了现有方法中可能存在的高频知识被低频知识淹没的问题。实验表明，该方法在六个图形数据集和三种常见的 GNN 架构中平均优于原始 MLPs 12.6％，并且优于对应的 GNN 2.6％。