Recently, Graph Transformers have emerged as a promising solution to alleviate the inherent limitations of Graph Neural Networks (GNNs) and enhance graph representation performance. Unfortunately, Graph Transformers are computationally expensive due to the quadratic complexity inherent in self-attention when applied over large-scale graphs, especially for node tasks. In contrast, spiking neural networks (SNNs), with event-driven and binary spikes properties, can perform energy-efficient computation. In this work, we propose a novel insight into integrating SNNs with Graph Transformers and design a Spiking Graph Attention (SGA) module. The matrix multiplication is replaced by sparse addition and mask operations. The linear complexity enables all-pair node interactions on large-scale graphs with limited GPU memory. To our knowledge, our work is the first attempt to introduce SNNs into Graph Transformers. Furthermore, we design SpikeGraphormer, a Dual-branch architecture, combining a sparse GNN branch with our SGA-driven Graph Transformer branch, which can simultaneously perform all-pair node interactions and capture local neighborhoods. SpikeGraphormer consistently outperforms existing state-of-the-art approaches across various datasets and makes substantial improvements in training time, inference time, and GPU memory cost (10 ~ 20x lower than vanilla self-attention). It also performs well in cross-domain applications (image and text classification). We release our code at https://github.com/PHD-lanyu/SpikeGraphormer.

我们提出了将SNNs与Graph Transformers相结合的新颖观点，并设计了一个Spiking Graph Attention (SGA)模块，通过稀疏加法和掩码操作取代矩阵乘法。我们的Spiking Graph Attention模块在大规模图上实现了全对节点交互，并且能够同时捕捉本地邻居。SpikeGraphormer比现有的方法在各种数据集上表现出色，在训练时间、推断时间和GPU内存成本等方面都有显著改进（比普通的self-attention低10-20倍）。它在跨领域应用（图像和文本分类）中也表现良好。

SpikeGraphormer：具有脉冲图注意力的高性能图Transformer