As the third-generation neural network, the Spiking Neural Network (SNN) has
the advantages of low power consumption and high energy efficiency, making it
suitable for implementation on edge devices. More recently, the most advanced
SNN, Spikformer, combines the self-attention module from Transformer with SNN
to achieve remarkable performance. However, it adopts larger channel dimensions
in MLP layers, leading to an increased number of redundant model parameters. To
effectively decrease the computational complexity and weight parameters of the
model, we explore the Lottery Ticket Hypothesis (LTH) and discover a very
sparse ($\ge$90%) subnetwork that achieves comparable performance to the
original network. Furthermore, we also design a lightweight token selector
module, which can remove unimportant background information from images based
on the average spike firing rate of neurons, selecting only essential
foreground image tokens to participate in attention calculation. Based on that,
we present SparseSpikformer, a co-design framework aimed at achieving sparsity
in Spikformer through token and weight pruning techniques. Experimental results
demonstrate that our framework can significantly reduce 90% model parameters
and cut down Giga Floating-Point Operations (GFLOPs) by 20% while maintaining
the accuracy of the original model.

SparseSpikformer 是一种通过令牌和权重修剪技术实现稀疏性的共设计框架，可以显著减少模型参数 90％并减少 20％的 GFLOPs，同时保持原始模型的准确性。

稀疏 Spikformer：脉冲变换器的令牌和权重剪枝的协同设计框架

SparseSpikformer: A Co-Design Framework for Token and Weight Pruning in  Spiking Transformer

Spiking neural networks (SNNs) offer a promising avenue to implement deep
neural networks in a more energy-efficient way. However, the network
architectures of existing SNNs for language tasks are too simplistic, and deep
architectures have not been fully explored, resulting in a significant
performance gap compared to mainstream transformer-based networks such as BERT.
To this end, we improve a recently-proposed spiking transformer (i.e.,
Spikformer) to make it possible to process language tasks and propose a
two-stage knowledge distillation method for training it, which combines
pre-training by distilling knowledge from BERT with a large collection of
unlabelled texts and fine-tuning with task-specific instances via knowledge
distillation again from the BERT fine-tuned on the same training examples.
Through extensive experimentation, we show that the models trained with our
method, named SpikeBERT, outperform state-of-the-art SNNs and even achieve
comparable results to BERTs on text classification tasks for both English and
Chinese with much less energy consumption.

我们改进了最近提出的脉冲变压器（即 Spikformer）以使其能够处理语言任务，并提出了一种两阶段知识蒸馏方法来训练它，该方法结合了通过从 BERT 对大量无标签文本进行知识蒸馏的预训练和通过再次从 BERT 在相同训练样本上进行知识蒸馏的微调来进行训练。通过广泛的实验证明，我们采用这种方法训练的模型，即 SpikeBERT，在英文和中文的文本分类任务上优于最先进的 SNNs，甚至能够达到与 BERTs 相当的结果，并且能够显著降低能量消耗。

SpikeBERT: 从 BERT 进行两阶段知识蒸馏训练的语言 Spikformer

SpikeBERT: A Language Spikformer Trained with Two-Stage Knowledge  Distillation from BERT

By integrating the self-attention capability and the biological properties of
Spiking Neural Networks (SNNs), Spikformer applies the flourishing Transformer
architecture to SNN design. It introduces a Spiking Self-Attention (SSA) module
to mix sparse visual features using spike-form Query, Key, and Value, resulting
in State-Of-The-Art (SOTA) performance on numerous datasets compared to
previous SNN-like frameworks. In this paper, we demonstrate that the Spikformer
architecture can be accelerated by replacing the SSA with an unparameterized
Linear Transform (LT) such as Fourier and Wavelet transforms. These transforms
are utilized to mix spike sequences, reducing the quadratic time complexity to
log-linear time complexity. They alternate between the frequency and time
domains to extract sparse visual features, showcasing powerful performance and
efficiency. We conduct extensive experiments on image classification using both
neuromorphic and static datasets. The results indicate that compared to the
SOTA Spikformer with SSA, Spikformer with LT achieves higher Top-1 accuracy on
neuromorphic datasets and comparable Top-1 accuracy on static datasets.
Moreover, Spikformer with LT achieves approximately $29$-$51\%$ improvement in
training speed, $61$-$70\%$ improvement in inference speed, and reduces memory
usage by $4$-$26\%$ due to not requiring learnable parameters.

通过将自注意力能力和脉冲神经网络（SNNs）的生物特性相结合，Spikformer 将蓬勃发展的 Transformer 架构应用于 SNN 设计。它引入了脉冲自注意力（SSA）模块，使用脉冲形式的查询、键和值来混合稀疏视觉特征，与之前的类 SNN 框架相比，在许多数据集上呈现出最先进的性能。该论文证明了 Spikformer 架构可以通过将 SSA 替换为非参数化的线性变换（LT），如傅里叶和小波变换，来加速。这些变换被用于混合脉冲序列，将二次时间复杂度降低为对数线性时间复杂度。它们在频率和时间域之间交替提取稀疏视觉特征，展示了强大的性能和效率。我们在使用神经形态学和静态数据集进行图像分类方面进行了广泛的实验。结果表明，与具有 SSA 的最先进的 Spikformer 相比，具有 LT 的 Spikformer 在神经形态学数据集上实现了更高的 Top-1 准确率，并且在静态数据集上实现了可比较的 Top-1 准确率。此外，与需要可学习参数的 SSA 相比，具有 LT 的 Spikformer 实现了约 29％至 51％的训练速度提升，61％至 70％的推断速度提升，并且减少了 4％至 26％的内存使用量。