Graph Convolutional Neural Networks (GCNs) possess strong capabilities for
processing graph data in non-grid domains. They can capture the topological
logical structure and node features in graphs and integrate them into nodes'
final representations. GCNs have been extensively studied in various fields,
such as recommendation systems, social networks, and protein molecular
structures. With the increasing application of graph neural networks, research
has focused on improving their performance while compressing their size. In
this work, a plug-in module named Graph Knowledge Enhancement and Distillation
Module (GKEDM) is proposed. GKEDM can enhance node representations and improve
the performance of GCNs by extracting and aggregating graph information via
multi-head attention mechanism. Furthermore, GKEDM can serve as an auxiliary
transferor for knowledge distillation. With a specially designed attention
distillation method, GKEDM can distill the knowledge of large teacher models
into high-performance and compact student models. Experiments on multiple
datasets demonstrate that GKEDM can significantly improve the performance of
various GCNs with minimal overhead. Furthermore, it can efficiently transfer
distilled knowledge from large teacher networks to small student networks via
attention distillation.

提出了一种名为图知识增强和蒸馏模块的插件，通过多头注意机制提取和聚合图信息以增强节点表示，并能通过特殊设计的注意力蒸馏方法将大型教师模型的知识蒸馏到高性能和紧凑的学生模型中，并通过注意力蒸馏有效地从大型教师网络转移蒸馏知识到小型学生网络。