DST methods achieve state-of-the-art results in sparse neural network
training, matching the generalization of dense models while enabling sparse
training and inference. Although the resulting models are highly sparse and
theoretically cheaper to train, achieving speedups with unstructured sparsity
on real-world hardware is challenging. In this work we propose a DST method to
learn a variant of structured N:M sparsity, the acceleration of which in
general is commonly supported in commodity hardware. Furthermore, we motivate
with both a theoretical analysis and empirical results, the generalization
performance of our specific N:M sparsity (constant fan-in), present a condensed
representation with a reduced parameter and memory footprint, and demonstrate
reduced inference time compared to dense models with a naive PyTorch CPU
implementation of the condensed representation Our source code is available at
this https URL

本研究提出了一种基于 DST 方法的 N:M 结构稀疏化学习方案，其具有较高的稀疏度并支持常见的硬件加速方式，在理论和实践中都表现出较好的泛化性能与较低的推理时间。

结构稀疏动态稀疏训练

Dynamic Sparse Training with Structured Sparsity

We compared the efficiency of the FlyHash model, an insect-inspired sparse
neural network (Dasgupta et al., 2017), to similar but non-sparse models in an
embodied navigation task. This requires a model to control steering by
comparing current visual inputs to memories stored along a training route. We
concluded the FlyHash model is more efficient than others, especially in terms
of data encoding.

本文比较了虫子灵感启发的稀疏神经网络 FlyHash 模型与类似但不稀疏的模型在体感导航任务中的效率，并得出结论：FlyHash 模型在数据编码方面尤其高效。

基于昆虫启发稀疏神经网络的视觉路线跟随

Vision-based route following by an embodied insect-inspired sparse  neural network

Recent advances in the sparse neural network literature have made it possible
to prune many large feed forward and convolutional networks with only a small
quantity of data. Yet, these same techniques often falter when applied to the
problem of recovering sparse recurrent networks. These failures are
quantitative: when pruned with recent techniques, RNNs typically obtain worse
performance than they do under a simple random pruning scheme. The failures are
also qualitative: the distribution of active weights in a pruned LSTM or GRU
network tend to be concentrated in specific neurons and gates, and not well
dispersed across the entire architecture. We seek to rectify both the
quantitative and qualitative issues with recurrent network pruning by
introducing a new recurrent pruning objective derived from the spectrum of the
recurrent Jacobian. Our objective is data efficient (requiring only 64 data
points to prune the network), easy to implement, and produces 95% sparse GRUs
that significantly improve on existing baselines. We evaluate on sequential
MNIST, Billion Words, and Wikitext.

这篇论文通过引入一种基于循环雅可比谱的新的循环网络修剪目标，提出了一种高效数据、易于实现的处理方案，能够解决循环网络修剪中的定量和定性问题，并在连续 MNIST、十亿词、Wikitext 等任务中取得了长足进展。

通过雅可比谱评估一次性剪枝递归神经网络

One-Shot Pruning of Recurrent Neural Networks by Jacobian Spectrum  Evaluation

We study the improper learning of multi-layer neural networks. Suppose that
the neural network to be learned has $k$ hidden layers and that the
$\ell_1$-norm of the incoming weights of any neuron is bounded by $L$. We
present a kernel-based method, such that with probability at least $1 -
\delta$, it learns a predictor whose generalization error is at most $\epsilon$
worse than that of the neural network. The sample complexity and the time
complexity of the presented method are polynomial in the input dimension and in
$(1/\epsilon,\log(1/\delta),F(k,L))$, where $F(k,L)$ is a function depending on
$(k,L)$ and on the activation function, independent of the number of neurons.
The algorithm applies to both sigmoid-like activation functions and ReLU-like
activation functions. It implies that any sufficiently sparse neural network is
learnable in polynomial time.

研究了多层神经网络的不当学习，提出了一种基于核的方法来学习预测器，其广义误差不超过神经网络的误差，时间复杂度为多项式级别.