As one of the energy-efficient alternatives of conventional neural networks
(CNNs), spiking neural networks (SNNs) have gained more and more interest
recently. To train the deep models, some effective batch normalization (BN)
techniques are proposed in SNNs. All these BNs are suggested to be used after
the convolution layer as usually doing in CNNs. However, the spiking neuron is
much more complex with the spatio-temporal dynamics. The regulated data flow
after the BN layer will be disturbed again by the membrane potential updating
operation before the firing function, i.e., the nonlinear activation.
Therefore, we advocate adding another BN layer before the firing function to
normalize the membrane potential again, called MPBN. To eliminate the induced
time cost of MPBN, we also propose a training-inference-decoupled
re-parameterization technique to fold the trained MPBN into the firing
threshold. With the re-parameterization technique, the MPBN will not introduce
any extra time burden in the inference. Furthermore, the MPBN can also adopt
the element-wised form, while these BNs after the convolution layer can only
use the channel-wised form. Experimental results show that the proposed MPBN
performs well on both popular non-spiking static and neuromorphic datasets. Our
code is open-sourced at \href{this https URL}{MPBN}.

本研究介绍了一种在脉冲神经网络中使用的新型批标准化方法，称为膜电位标准化（MPBN），以优化深层模型的训练过程，并通过重新参数化技术将其与脉冲神经网络的推断步骤解耦，实现有效的时间性能。

膜电位批归一化用于脉冲神经网络

Membrane Potential Batch Normalization for Spiking Neural Networks

The Spiking Neural Network (SNN) has attracted more and more attention
recently. It adopts binary spike signals to transmit information. Benefitting
from the information passing paradigm of SNNs, the multiplications of
activations and weights can be replaced by additions, which are more
energy-efficient. However, its ``Hard Reset" mechanism for the firing activity
would ignore the difference among membrane potentials when the membrane
potential is above the firing threshold, causing information loss. Meanwhile,
quantifying the membrane potential to 0/1 spikes at the firing instants will
inevitably introduce the quantization error thus bringing about information
loss too. To address these problems, we propose to use the ``Soft Reset"
mechanism for the supervised training-based SNNs, which will drive the membrane
potential to a dynamic reset potential according to its magnitude, and Membrane
Potential Rectifier (MPR) to reduce the quantization error via redistributing
the membrane potential to a range close to the spikes. Results show that the
SNNs with the ``Soft Reset" mechanism and MPR outperform their vanilla
counterparts on both static and dynamic datasets.

使用 “软复位” 机制和膜电位整流器 (MPR) 的脉冲神经网络 (SNNs) 在静态和动态数据集上表现优于普通 SNNs。

降低脉冲神经网络的信息损失

Reducing Information Loss for Spiking Neural Networks

Spiking Neural Networks (SNNs) have been attached great importance due to the
distinctive properties of low power consumption, biological plausibility, and
adversarial robustness. The most effective way to train deep SNNs is through
ANN-to-SNN conversion, which have yielded the best performance in deep network
structure and large-scale datasets. However, there is a trade-off between
accuracy and latency. In order to achieve high precision as original ANNs, a
long simulation time is needed to match the firing rate of a spiking neuron
with the activation value of an analog neuron, which impedes the practical
application of SNN. In this paper, we aim to achieve high-performance converted
SNNs with extremely low latency (fewer than 32 time-steps). We start by
theoretically analyzing ANN-to-SNN conversion and show that scaling the
thresholds does play a similar role as weight normalization. Instead of
introducing constraints that facilitate ANN-to-SNN conversion at the cost of
model capacity, we applied a more direct way by optimizing the initial membrane
potential to reduce the conversion loss in each layer. Besides, we demonstrate
that optimal initialization of membrane potentials can implement expected
error-free ANN-to-SNN conversion. We evaluate our algorithm on the CIFAR-10,
CIFAR-100 and ImageNet datasets and achieve state-of-the-art accuracy, using
fewer time-steps. For example, we reach top-1 accuracy of 93.38\% on CIFAR-10
with 16 time-steps. Moreover, our method can be applied to other ANN-SNN
conversion methodologies and remarkably promote performance when the time-steps
is small.

通过优化初始膜电位实现高精度 ANN-to-SNN 转换，使得 SNN 在低延迟下达到了与最优性能相当的准确率，并在 CIFAR-10、CIFAR-100 和 ImageNet 数据集上取得了最新的结果。

低延迟脉冲神经网络的优化电位初始化

Optimized Potential Initialization for Low-latency Spiking Neural  Networks

Nowadays deep learning is dominating the field of machine learning with
state-of-the-art performance in various application areas. Recently, spiking
neural networks (SNNs) have been attracting a great deal of attention, notably
owning to their power efficiency, which can potentially allow us to implement a
low-power deep learning engine suitable for real-time/mobile applications.
However, implementing SNN-based deep learning remains challenging, especially
gradient-based training of SNNs by error backpropagation. We cannot simply
propagate errors through SNNs in conventional way because of the property of
SNNs that process discrete data in the form of a series. Consequently, most of
the previous studies employ a workaround technique, which first trains a
conventional weighted-sum deep neural network and then maps the learning
weights to the SNN under training, instead of training SNN parameters directly.
In order to eliminate this workaround, recently proposed is a new class of SNN
named deep spiking networks (DSNs), which can be trained directly (without a
mapping from conventional deep networks) by error backpropagation with
stochastic gradient descent. In this paper, we show that the initialization of
the membrane potential on the backward path is an important step in DSN
training, through diverse experiments performed under various conditions.
Furthermore, we propose a simple and efficient method that can improve DSN
training by controlling the initial membrane potential on the backward path. In
our experiments, adopting the proposed approach allowed us to boost the
performance of DSN training in terms of converging time and accuracy.

本文介绍了一种新型的深度脉冲神经网络（DSN），它可以直接进行参数训练，不需要先用传统深度神经网络的训练结果来初始化；研究表明，在 DSN 中控制反向路径的膜电位初始值是非常重要的，同时本文提出了一种简单而有效的方法可以加快 DSN 的收敛时间并提高准确性。