As the size of large language models continue to scale, so does the
computational resources required to run it. Spiking neural networks (SNNs) have
emerged as an energy-efficient approach to deep learning that leverage sparse
and event-driven activations to reduce the computational overhead associated
with model inference. While they have become competitive with non-spiking
models on many computer vision tasks, SNNs have also proven to be more
challenging to train. As a result, their performance lags behind modern deep
learning, and we are yet to see the effectiveness of SNNs in language
generation. In this paper, inspired by the RWKV language model, we successfully
implement `SpikeGPT', a generative language model with pure binary,
event-driven spiking activation units. We train the proposed model on three
model variants: 45M, 125M and 260M parameters. To the best of our knowledge,
this is 4x larger than any functional backprop-trained SNN to date. We achieve
this by modifying the transformer block to replace multi-head self attention to
reduce quadratic computational complexity to linear with increasing sequence
length. Input tokens are instead streamed in sequentially to our attention
mechanism (as with typical SNNs). Our preliminary experiments show that
SpikeGPT remains competitive with non-spiking models on tested benchmarks,
while maintaining 5x less energy consumption when processed on neuromorphic
hardware that can leverage sparse, event-driven activations. Our code
implementation is available at this https URL.

本文提出了基于 SNN 的生成语言模型 SpikeGPT，利用二进制、SNN 的事件驱动稀疏激活单元，将传统的注意力机制修改为逐步输入机制，并在多个数据集上验证了其性能与能耗，结果表明 SpikeGPT 在处理自然语言生成任务时表现出色并且能耗显著降低。