Spiking Neural Networks (SNNs), known for their biologically plausible
architecture, face the challenge of limited performance. The self-attention
mechanism, which is the cornerstone of the high-performance Transformer and
also a biologically inspired structure, is absent in existing SNNs. To this
end, we explore the potential of leveraging both self-attention capability and
biological properties of SNNs, and propose a novel Spiking Self-Attention (SSA)
and Spiking Transformer (Spikformer). The SSA mechanism eliminates the need for
softmax and captures the sparse visual feature employing spike-based Query,
Key, and Value. This sparse computation without multiplication makes SSA
efficient and energy-saving. Further, we develop a Spiking Convolutional Stem
(SCS) with supplementary convolutional layers to enhance the architecture of
Spikformer. The Spikformer enhanced with the SCS is referred to as Spikformer
V2. To train larger and deeper Spikformer V2, we introduce a pioneering
exploration of Self-Supervised Learning (SSL) within the SNN. Specifically, we
pre-train Spikformer V2 with masking and reconstruction style inspired by the
mainstream self-supervised Transformer, and then finetune the Spikformer V2 on
the image classification on ImageNet. Extensive experiments show that
Spikformer V2 outperforms other previous surrogate training and ANN2SNN
methods. An 8-layer Spikformer V2 achieves an accuracy of 80.38% using 4 time
steps, and after SSL, a 172M 16-layer Spikformer V2 reaches an accuracy of
81.10% with just 1 time step. To the best of our knowledge, this is the first
time that the SNN achieves 80+% accuracy on ImageNet. The code will be
available at Spikformer V2.

通过结合自注意能力和生物特性，本论文提出了一种新颖的脉冲自注意 (SSA) 和脉冲变换器 (Spikformer)，其中 SSA 机制消除了 softmax 的需求，利用基于脉冲的查询、键和值实现了稀疏视觉特征的捕获，而 SCS 则用于增强 Spikformer 的架构。通过自监督学习 (SLS) 实现对更大、更深的 Spikformer V2 的训练，实验证明此方法在 ImageNet 上的分类准确性超过了其他方法，且 SNN 首次在 ImageNet 上达到超过 80% 的准确性。