Neuromorphic computing, which exploits Spiking Neural Networks (SNNs) on
neuromorphic chips, is a promising energy-efficient alternative to traditional
AI. CNN-based SNNs are the current mainstream of neuromorphic computing. By
contrast, no neuromorphic chips are designed especially for Transformer-based
SNNs, which have just emerged, and their performance is only on par with
CNN-based SNNs, offering no distinct advantage. In this work, we propose a
general Transformer-based SNN architecture, termed as ``Meta-SpikeFormer",
whose goals are: 1) Lower-power, supports the spike-driven paradigm that there
is only sparse addition in the network; 2) Versatility, handles various vision
tasks; 3) High-performance, shows overwhelming performance advantages over
CNN-based SNNs; 4) Meta-architecture, provides inspiration for future
next-generation Transformer-based neuromorphic chip designs. Specifically, we
extend the Spike-driven Transformer in \citet{yao2023spike} into a meta
architecture, and explore the impact of structure, spike-driven self-attention,
and skip connection on its performance. On ImageNet-1K, Meta-SpikeFormer
achieves 80.0\% top-1 accuracy (55M), surpassing the current state-of-the-art
(SOTA) SNN baselines (66M) by 3.7\%. This is the first direct training SNN
backbone that can simultaneously supports classification, detection, and
segmentation, obtaining SOTA results in SNNs. Finally, we discuss the
inspiration of the meta SNN architecture for neuromorphic chip design. Source
code and models are available at
https://github.com/BICLab/Spike-Driven-Transformer-V2.

通过提出一种名为 "Meta-SpikeFormer" 的通用 Transformer 架构，本论文实现了对 CNN-based SNNs 具有压倒性性能优势的低功耗、多功能、高性能、面向未来下一代神经形态芯片设计的 Meta 架构，并应用于分类、检测和分割任务中，取得了 SNN 领域的最佳结果。

脉冲驱动 Transformer V2：元脉冲神经网络架构启发下一代神经形态芯片的设计

Spike-driven Transformer V2: Meta Spiking Neural Network Architecture  Inspiring the Design of Next-generation Neuromorphic Chips

Spiking Neural Networks (SNNs) provide an energy-efficient deep learning
option due to their unique spike-based event-driven (i.e., spike-driven)
paradigm. In this paper, we incorporate the spike-driven paradigm into
Transformer by the proposed Spike-driven Transformer with four unique
properties: 1) Event-driven, no calculation is triggered when the input of
Transformer is zero; 2) Binary spike communication, all matrix multiplications
associated with the spike matrix can be transformed into sparse additions; 3)
Self-attention with linear complexity at both token and channel dimensions; 4)
The operations between spike-form Query, Key, and Value are mask and addition.
Together, there are only sparse addition operations in the Spike-driven
Transformer. To this end, we design a novel Spike-Driven Self-Attention (SDSA),
which exploits only mask and addition operations without any multiplication,
and thus having up to $87.2\times$ lower computation energy than vanilla
self-attention. Especially in SDSA, the matrix multiplication between Query,
Key, and Value is designed as the mask operation. In addition, we rearrange all
residual connections in the vanilla Transformer before the activation functions
to ensure that all neurons transmit binary spike signals. It is shown that the
Spike-driven Transformer can achieve 77.1\% top-1 accuracy on ImageNet-1K,
which is the state-of-the-art result in the SNN field. The source code is
available at this https URL

本文提出了一种基于 Spike-driven Transformer 的神经网络，其具有事件驱动、自注意力、二进制尖峰通讯等特点，从而实现了能效更高的深度学习，并在 ImageNet-1K 数据集上取得了 77.1% 的 top-1 准确率。