Spiking neural networks (SNNs) have superb characteristics in sensory
information recognition tasks due to their biological plausibility. However,
the performance of some current spiking-based models is limited by their
structures which means either fully connected or too-deep structures bring too
much redundancy. This redundancy from both connection and neurons is one of the
key factors hindering the practical application of SNNs. Although Some pruning
methods were proposed to tackle this problem, they normally ignored the fact
the neural topology in the human brain could be adjusted dynamically. Inspired
by this, this paper proposed an evolutionary-based structure construction
method for constructing more reasonable SNNs. By integrating the knowledge
distillation and connection pruning method, the synaptic connections in SNNs
can be optimized dynamically to reach an optimal state. As a result, the
structure of SNNs could not only absorb knowledge from the teacher model but
also search for deep but sparse network topology. Experimental results on
CIFAR100 and DVS-Gesture show that the proposed structure learning method can
get pretty well performance while reducing the connection redundancy. The
proposed method explores a novel dynamical way for structure learning from
scratch in SNNs which could build a bridge to close the gap between deep
learning and bio-inspired neural dynamics.

本文提出了一种基于进化算法的结构构建方法，通过集成知识蒸馏和连接修剪方法，优化 SNN 的突触连接以达到最优状态，并在 CIFAR100 和 DVS-Geste 上进行了实验，结果表明该方法可以在降低连接冗余的同时获得良好的性能。

基于反向知识蒸馏的生物启发式结构学习与脉冲神经网络

Biologically inspired structure learning with reverse knowledge  distillation for spiking neural networks

As well known, the huge memory and compute costs of both artificial neural
networks (ANNs) and spiking neural networks (SNNs) greatly hinder their
deployment on edge devices with high efficiency. Model compression has been
proposed as a promising technique to improve the running efficiency via
parameter and operation reduction. Whereas, this technique is mainly practiced
in ANNs rather than SNNs. It is interesting to answer how much an SNN model can
be compressed without compromising its functionality, where two challenges
should be addressed: i) the accuracy of SNNs is usually sensitive to model
compression, which requires an accurate compression methodology; ii) the
computation of SNNs is event-driven rather than static, which produces an extra
compression dimension on dynamic spikes. To this end, we realize a
comprehensive SNN compression through three steps. First, we formulate the
connection pruning and weight quantization as a constrained optimization
problem. Second, we combine spatio-temporal backpropagation (STBP) and
alternating direction method of multipliers (ADMM) to solve the problem with
minimum accuracy loss. Third, we further propose activity regularization to
reduce the spike events for fewer active operations. These methods can be
applied in either a single way for moderate compression or a joint way for
aggressive compression. We define several quantitative metrics to evaluation
the compression performance for SNNs. Our methodology is validated in pattern
recognition tasks over MNIST, N-MNIST, CIFAR10, and CIFAR100 datasets, where
extensive comparisons, analyses, and insights are provided. To our best
knowledge, this is the first work that studies SNN compression in a
comprehensive manner by exploiting all compressible components and achieves
better results.

对神经元网络进行压缩，能够在不影响功能的情况下提高运行效率。文章提出了一种深度学习模型压缩技术，通过针对网络的每个可压缩部分，实现神经元模型在 MNIST、N-MNIST、CIFAR10 和 CIFAR100 数据集上的模式识别任务，并取得了更好的结果。

使用 ADMM 优化和活动规范化的综合 SNN 压缩

Comprehensive SNN Compression Using ADMM Optimization and Activity  Regularization

Deep learning has become a ubiquitous technology to improve machine
intelligence. However, most of the existing deep models are structurally very
complex, making them difficult to be deployed on the mobile platforms with
limited computational power. In this paper, we propose a novel network
compression method called dynamic network surgery, which can remarkably reduce
the network complexity by making on-the-fly connection pruning. Unlike the
previous methods which accomplish this task in a greedy way, we properly
incorporate connection splicing into the whole process to avoid incorrect
pruning and make it as a continual network maintenance. The effectiveness of
our method is proved with experiments. Without any accuracy loss, our method
can efficiently compress the number of parameters in LeNet-5 and AlexNet by a
factor of $\bm{108}\times$ and $\bm{17.7}\times$ respectively, proving that it
outperforms the recent pruning method by considerable margins. Code and some
models are available at this https URL

本研究提出了一种名为动态网络手术的网络压缩方法，能够通过即时连接修剪显著降低网络复杂性，同时避免了错误裁剪的问题，并将其作为一个持续的网络维护过程，实验结果表明，该方法能够有效地压缩 LeNet-5 和 AlexNet 的参数数量，而不会损失精度，同时优于最近的裁剪方法。