Energy-efficient spikformer has been proposed by integrating the biologically
plausible spiking neural network (SNN) and artificial Transformer, whereby the
Spiking Self-Attention (SSA) is used to achieve both higher accuracy and lower
computational cost. However, it seems that self-attention is not always
necessary, especially in sparse spike-form calculation manners. In this paper,
we innovatively replace vanilla SSA (using dynamic bases calculating from Query
and Key) with spike-form Fourier Transform, Wavelet Transform, and their
combinations (using fixed triangular or wavelets bases), based on a key
hypothesis that both of them use a set of basis functions for information
transformation. Hence, the Fourier-or-Wavelet-based spikformer (FWformer) is
proposed and verified in visual classification tasks, including both static
image and event-based video datasets. The FWformer can achieve comparable or
even higher accuracies ($0.4\%$-$1.5\%$), higher running speed ($9\%$-$51\%$
for training and $19\%$-$70\%$ for inference), reduced theoretical energy
consumption ($20\%$-$25\%$), and reduced GPU memory usage ($4\%$-$26\%$),
compared to the standard spikformer. Our result indicates the continuous
refinement of new Transformers, that are inspired either by biological
discovery (spike-form), or information theory (Fourier or Wavelet Transform),
is promising.

使用峰值傅立叶变换、小波变换及其组合替代传统的自注意力机制，提出了基于峰值或小波的 spikformer（FWformer），在视觉分类任务中验证并表明其能够达到可比甚至更高的准确性（0.4%-1.5%），更快的运行速度（训练速度提高 9%-51%，推理速度提高 19%-70%），降低的理论能耗（降低 20%-25%），以及降低的 GPU 内存使用（降低 4%-26%），相比于标准的 spikformer。结果显示，基于生物发现（峰值形式）或信息理论（傅立叶或小波变换）的新 Transformer 的持续改进是值得期待的。

用于高效视觉分类的 Spikformer 中傅立叶或小波基作为对应的自注意力

Fourier or Wavelet bases as counterpart self-attention in spikformer for  efficient visual classification

Spiking Neural Networks (SNNs), known for their biologically plausible
architecture, face the challenge of limited performance. The self-attention
mechanism, which is the cornerstone of the high-performance Transformer and
also a biologically inspired structure, is absent in existing SNNs. To this
end, we explore the potential of leveraging both self-attention capability and
biological properties of SNNs, and propose a novel Spiking Self-Attention (SSA)
and Spiking Transformer (Spikformer). The SSA mechanism eliminates the need for
softmax and captures the sparse visual feature employing spike-based Query,
Key, and Value. This sparse computation without multiplication makes SSA
efficient and energy-saving. Further, we develop a Spiking Convolutional Stem
(SCS) with supplementary convolutional layers to enhance the architecture of
Spikformer. The Spikformer enhanced with the SCS is referred to as Spikformer
V2. To train larger and deeper Spikformer V2, we introduce a pioneering
exploration of Self-Supervised Learning (SSL) within the SNN. Specifically, we
pre-train Spikformer V2 with masking and reconstruction style inspired by the
mainstream self-supervised Transformer, and then finetune the Spikformer V2 on
the image classification on ImageNet. Extensive experiments show that
Spikformer V2 outperforms other previous surrogate training and ANN2SNN
methods. An 8-layer Spikformer V2 achieves an accuracy of 80.38% using 4 time
steps, and after SSL, a 172M 16-layer Spikformer V2 reaches an accuracy of
81.10% with just 1 time step. To the best of our knowledge, this is the first
time that the SNN achieves 80+% accuracy on ImageNet. The code will be
available at Spikformer V2.

通过结合自注意能力和生物特性，本论文提出了一种新颖的脉冲自注意 (SSA) 和脉冲变换器 (Spikformer)，其中 SSA 机制消除了 softmax 的需求，利用基于脉冲的查询、键和值实现了稀疏视觉特征的捕获，而 SCS 则用于增强 Spikformer 的架构。通过自监督学习 (SLS) 实现对更大、更深的 Spikformer V2 的训练，实验证明此方法在 ImageNet 上的分类准确性超过了其他方法，且 SNN 首次在 ImageNet 上达到超过 80% 的准确性。