Towards energy-efficient artificial intelligence similar to the human brain, the bio-inspired spiking neural networks (SNNs) have advantages of biological plausibility, event-driven sparsity, and binary activation. Recently, large-scale language models exhibit promising generalization capability, making it a valuable issue to explore more general spike-driven models. However, the binary spikes in existing SNNs fail to encode adequate semantic information, placing technological challenges for generalization. This work proposes the first fully spiking mechanism for general language tasks, including both discriminative and generative ones. Different from previous spikes with {0,1} levels, we propose a more general spike formulation with bi-directional, elastic amplitude, and elastic frequency encoding, while still maintaining the addition nature of SNNs. In a single time step, the spike is enhanced by direction and amplitude information; in spike frequency, a strategy to control spike firing rate is well designed. We plug this elastic bi-spiking mechanism in language modeling, named SpikeLM. It is the first time to handle general language tasks with fully spike-driven models, which achieve much higher accuracy than previously possible. SpikeLM also greatly bridges the performance gap between SNNs and ANNs in language modeling. Our code is available at https://github.com/Xingrun-Xing/SpikeLM.

向类似人脑的高能效人工智能迈进，生物启发的脉冲神经网络（SNN）具有生物合理性、事件驱动的稀疏性和二值激活的优势。本文提出了首个全脉冲机制，以处理包括鉴别性和生成性任务在内的一般性语言任务。与现有SNN中的二值脉冲不同，我们提出了一种具有双向、弹性幅度和弹性频率编码的更一般的脉冲形式，同时仍然保持了SNN的加法性质。在单个时间步内，脉冲通过方向和幅度信息增强；在脉冲频率方面，我们设计了一种控制脉冲发射率的策略。我们将这种弹性双脉冲机制应用于语言建模，命名为SpikeLM。这是首次使用全脉冲驱动模型处理一般性语言任务，其准确性远高于以前可能的水平。SpikeLM还极大地弥合了SNN和ANN在语言建模方面的性能差距。我们的代码可在此网址上获得：https://此URL。

SpikeLM: 基于弹性双脉冲机制的通用脉冲驱动语言建模