Although the convolutional neural network (CNN) has achieved excellent
performance in vision tasks by extracting the intra-sample representation, it
will take a higher training expense because of stacking numerous convolutional
layers. Recently, as the bilinear models, graph neural networks (GNN) have
succeeded in exploring the underlying topological relationship among the graph
data with a few graph neural layers. Unfortunately, it cannot be directly
utilized on non-graph data due to the lack of graph structure and has high
inference latency on large-scale scenarios. Inspired by these complementary
strengths and weaknesses, \textit{we discuss a natural question, how to bridge
these two heterogeneous networks?} In this paper, we propose a novel CNN2GNN
framework to unify CNN and GNN together via distillation. Firstly, to break the
limitations of GNN, a differentiable sparse graph learning module is designed
as the head of networks to dynamically learn the graph for inductive learning.
Then, a response-based distillation is introduced to transfer the knowledge
from CNN to GNN and bridge these two heterogeneous networks. Notably, due to
extracting the intra-sample representation of a single instance and the
topological relationship among the datasets simultaneously, the performance of
distilled ``boosted'' two-layer GNN on Mini-ImageNet is much higher than CNN
containing dozens of layers such as ResNet152.

基于卷积神经网络（CNN）和图神经网络（GNN）的互补优势和弱点，本文提出了一种新颖的 CNN2GNN 框架，通过知识蒸馏将 CNN 和 GNN 结合起来。