Inducing and leveraging sparse activations during training and inference is a
promising avenue for improving the computational efficiency of deep networks,
which is increasingly important as network sizes continue to grow and their
application becomes more widespread. Here we use the large width Gaussian
process limit to analyze the behaviour, at random initialization, of nonlinear
activations that induce sparsity in the hidden outputs. A previously unreported
form of training instability is proven for arguably two of the most natural
candidates for hidden layer sparsification; those being a shifted ReLU
($\phi(x)=\max(0, x-\tau)$ for $\tau\ge 0$) and soft thresholding ($\phi(x)=0$
for $|x|\le\tau$ and $x-\text{sign}(x)\tau$ for $|x|>\tau$). We show that this
instability is overcome by clipping the nonlinear activation magnitude, at a
level prescribed by the shape of the associated Gaussian process variance map.
Numerical experiments verify the theory and show that the proposed magnitude
clipped sparsifying activations can be trained with training and test
fractional sparsity as high as 85\% while retaining close to full accuracy.

通过剪枝层来诱导和利用稀疏激活是提高深度网络计算效率的一种有前途的方法，本论文使用大尺度高斯过程极限分析了随机初始化时诱导隐藏层稀疏性的非线性激活函数，证明了一种先前未报告的培训不稳定性，并表明通过剪枝激活函数的幅度，可以克服这种不稳定性，理论验证和数值实验表明，这种剪枝激活函数能够在训练和测试时保持接近完全准确度的同时达到高达 85％的稀疏度。

稀疏诱导激活的深度神经网络初始化

Deep Neural Network Initialization with Sparsity Inducing Activations

Artificial neural networks open up unprecedented machine learning
capabilities at the cost of ever growing computational requirements.
Sparsifying the parameters, often achieved through weight pruning, has been
identified as a powerful technique to compress the number of model parameters
and reduce the computational operations of neural networks. Yet, sparse
activations, while omnipresent in both biological neural networks and deep
learning systems, have not been fully utilized as a compression technique in
deep learning. Moreover, the interaction between sparse activations and weight
pruning is not fully understood. In this work, we demonstrate that activity
sparsity can compose multiplicatively with parameter sparsity in a recurrent
neural network model based on the GRU that is designed to be activity sparse.
We achieve up to $20\times$ reduction of computation while maintaining
perplexities below $60$ on the Penn Treebank language modeling task. This
magnitude of reduction has not been achieved previously with solely sparsely
connected LSTMs, and the language modeling performance of our model has not
been achieved previously with any sparsely activated recurrent neural networks
or spiking neural networks. Neuromorphic computing devices are especially good
at taking advantage of the dynamic activity sparsity, and our results provide
strong evidence that making deep learning models activity sparse and porting
them to neuromorphic devices can be a viable strategy that does not compromise
on task performance. Our results also drive further convergence of methods from
deep learning and neuromorphic computing for efficient machine learning.

通过对参数进行稀疏化，结合稀疏激活在递归神经网络中的相乘作用，实现了对神经网络计算成本高效率的优化，可以应用于神经形态计算设备上，同时不牺牲任务性能。

活动稀疏性与权重稀疏性相辅相成，用于高效的 RNN 推理

Activity Sparsity Complements Weight Sparsity for Efficient RNN  Inference

Deep neural networks lack straightforward ways to incorporate domain
knowledge and are notoriously considered black boxes. Prior works attempted to
inject domain knowledge into architectures implicitly through data
augmentation. Building on recent advances on equivariant neural networks, we
propose networks that explicitly encode domain knowledge, specifically
equivariance with respect to rotations. By using unfolded architectures, a rich
framework that originated from sparse coding and has theoretical guarantees, we
present interpretable networks with sparse activations. The equivariant
unfolded networks compete favorably with baselines, with only a fraction of
their parameters, as showcased on (rotated) MNIST and CIFAR-10.

本文介绍了一种利用等变神经网络显性地加入领域知识的框架，该框架基于稀疏编码，以旋转等变性为例。通过在（旋转）MNIST 和 CIFAR-10 上的表现，得出了具有稀疏激活的可解释性网络与基线网络相竞争的结论。