Activity and parameter sparsity are two standard methods of making neural
networks computationally more efficient. Event-based architectures such as
spiking neural networks (SNNs) naturally exhibit activity sparsity, and many
methods exist to sparsify their connectivity by pruning weights. While the
effect of weight pruning on feed-forward SNNs has been previously studied for
computer vision tasks, the effects of pruning for complex sequence tasks like
language modeling are less well studied since SNNs have traditionally struggled
to achieve meaningful performance on these tasks. Using a recently published
SNN-like architecture that works well on small-scale language modeling, we
study the effects of weight pruning when combined with activity sparsity.
Specifically, we study the trade-off between the multiplicative efficiency
gains the combination affords and its effect on task performance for language
modeling. To dissect the effects of the two sparsities, we conduct a
comparative analysis between densely activated models and sparsely activated
event-based models across varying degrees of connectivity sparsity. We
demonstrate that sparse activity and sparse connectivity complement each other
without a proportional drop in task performance for an event-based neural
network trained on the Penn Treebank and WikiText-2 language modeling datasets.
Our results suggest sparsely connected event-based neural networks are
promising candidates for effective and efficient sequence modeling.

通过研究神经网络的活动稀疏性和参数稀疏性，探讨了在序列建模中将两者相结合的权重剪枝的效果，结果表明稀疏连接的基于事件的神经网络是有效和高效的序列建模模型。

神经形态语言模型中的权重稀疏性与活动稀疏性相辅相成

Weight Sparsity Complements Activity Sparsity in Neuromorphic Language  Models

Backpropagation through time (BPTT) is the standard algorithm for training
recurrent neural networks (RNNs), which requires separate simulation phases for
the forward and backward passes for inference and learning, respectively.
Moreover, BPTT requires storing the complete history of network states between
phases, with memory consumption growing proportional to the input sequence
length. This makes BPTT unsuited for online learning and presents a challenge
for implementation on low-resource real-time systems. Real-Time Recurrent
Learning (RTRL) allows online learning, and the growth of required memory is
independent of sequence length. However, RTRL suffers from exceptionally high
computational costs that grow proportional to the fourth power of the state
size, making RTRL computationally intractable for all but the smallest of
networks. In this work, we show that recurrent networks exhibiting high
activity sparsity can reduce the computational cost of RTRL. Moreover,
combining activity and parameter sparsity can lead to significant enough
savings in computational and memory costs to make RTRL practical. Unlike
previous work, this improvement in the efficiency of RTRL can be achieved
without using any approximations for the learning process.

本文提出，对于循环网络，高效的实时递归学习可以通过结合活动稀疏性和参数稀疏性得到显着的计算和存储成本节省，无需使用任何近似来学习过程。