Spiking Neural Networks (SNNs) have recently emerged as a new generation of
low-power deep neural networks, which is suitable to be implemented on
low-power mobile/edge devices. As such devices have limited memory storage,
neural pruning on SNNs has been widely explored in recent years. Most existing
SNN pruning works focus on shallow SNNs (2~6 layers), however, deeper SNNs (>16
layers) are proposed by state-of-the-art SNN works, which is difficult to be
compatible with the current SNN pruning work. To scale up a pruning technique
towards deep SNNs, we investigate Lottery Ticket Hypothesis (LTH) which states
that dense networks contain smaller subnetworks (i.e., winning tickets) that
achieve comparable performance to the dense networks. Our studies on LTH reveal
that the winning tickets consistently exist in deep SNNs across various
datasets and architectures, providing up to 97% sparsity without huge
performance degradation. However, the iterative searching process of LTH brings
a huge training computational cost when combined with the multiple timesteps of
SNNs. To alleviate such heavy searching cost, we propose Early-Time (ET) ticket
where we find the important weight connectivity from a smaller number of
timesteps. The proposed ET ticket can be seamlessly combined with a common
pruning techniques for finding winning tickets, such as Iterative Magnitude
Pruning (IMP) and Early-Bird (EB) tickets. Our experiment results show that the
proposed ET ticket reduces search time by up to 38% compared to IMP or EB
methods. Code is available at Github.

该研究使用彩票票据原理，提出了一种适用于深度脉冲神经网络（SNN）的神经剪枝技术，即运用早期时间（ET）票据来减少彩票票据原理的搜索时间，该方法在不降低性能的情况下使得 SNN 的鲁棒性得到了极大提高。

探索突触传递网络中的抽奖票假设

Exploring Lottery Ticket Hypothesis in Spiking Neural Networks

Previous work showed empirically that large neural networks can be
significantly reduced in size while preserving their accuracy. Model
compression became a central research topic, as it is crucial for deployment of
neural networks on devices with limited computational and memory resources. The
majority of the compression methods are based on heuristics and offer no
worst-case guarantees on the trade-off between the compression rate and the
approximation error for an arbitrarily new sample. We propose the first
efficient, data-independent neural pruning algorithm with a provable trade-off
between its compression rate and the approximation error for any future test
sample. Our method is based on the coreset framework, which finds a small
weighted subset of points that provably approximates the original inputs.
Specifically, we approximate the output of a layer of neurons by a coreset of
neurons in the previous layer and discard the rest. We apply this framework in
a layer-by-layer fashion from the top to the bottom. Unlike previous works, our
coreset is data independent, meaning that it provably guarantees the accuracy
of the function for any input $x\in \mathbb{R}^d$, including an adversarial
one. We demonstrate the effectiveness of our method on popular network
architectures. In particular, our coresets yield 90\% compression of the
LeNet-300-100 architecture on MNIST while improving the accuracy.

这篇论文提出了一种基于 coreset 方法的神经剪枝算法，可在不影响准确性的情况下压缩神经网络，并在 MNIST 数据集上进行了实验。

基于核集的数据独立神经剪枝

Data-Independent Neural Pruning via Coresets

We introduce Continual Learning via Neural Pruning (CLNP), a new method aimed
at lifelong learning in fixed capacity models based on neuronal model
sparsification. In this method, subsequent tasks are trained using the inactive
neurons and filters of the sparsified network and cause zero deterioration to
the performance of previous tasks. In order to deal with the possible
compromise between model sparsity and performance, we formalize and incorporate
the concept of graceful forgetting: the idea that it is preferable to suffer a
small amount of forgetting in a controlled manner if it helps regain network
capacity and prevents uncontrolled loss of performance during the training of
future tasks. CLNP also provides simple continual learning diagnostic tools in
terms of the number of free neurons left for the training of future tasks as
well as the number of neurons that are being reused. In particular, we see in
experiments that CLNP verifies and automatically takes advantage of the fact
that the features of earlier layers are more transferable. We show empirically
that CLNP leads to significantly improved results over current weight
elasticity based methods.

本篇研究提出了一种名为 CLNP 的方法，通过神经模型稀疏化实现模型固定容量下的全生命周期学习，在这种方法中，使用经过稀疏化网络中的非活动神经元和滤波器来训练后续任务，并对以前任务的性能不会造成任何恶化，此外，CLNP 还提供了简单的学习诊断工具。实验证明，与当前基于权重弹性的方法相比，CLNP 能够显著提高结果。