The sizes of deep neural networks (DNNs) are rapidly outgrowing the capacity
of hardware to store and train them. Research over the past few decades has
explored the prospect of sparsifying DNNs before, during, and after training by
pruning edges from the underlying topology. The resulting neural network is
known as a sparse neural network. More recent work has demonstrated the
remarkable result that certain sparse DNNs can train to the same precision as
dense DNNs at lower runtime and storage cost. An intriguing class of these
sparse DNNs is the X-Nets, which are initialized and trained upon a sparse
topology with neither reference to a parent dense DNN nor subsequent pruning.
We present an algorithm that deterministically generates RadiX-Nets: sparse DNN
topologies that, as a whole, are much more diverse than X-Net topologies, while
preserving X-Nets' desired characteristics. We further present a
functional-analytic conjecture based on the longstanding observation that
sparse neural network topologies can attain the same expressive power as dense
counterparts

本研究提出一种算法，可以确定生成 RadiX-Nets，这些稀疏的深度神经网络拓扑结构相比 X-Net 拓扑结构更加多样化，同时保留了 X-Nets 所需的特性，基于长期观察到的稀疏神经网络可以达到与稠密对应物相同的表达能力的功能分析猜想。

RadiX-Net: 深度神经网络的结构稀疏矩阵

RadiX-Net: Structured Sparse Matrices for Deep Neural Networks

Efficient CNN designs like ResNets and DenseNet were proposed to improve
accuracy vs efficiency trade-offs. They essentially increased the connectivity,
allowing efficient information flow across layers. Inspired by these
techniques, we propose to model connections between filters of a CNN using
graphs which are simultaneously sparse and well connected. Sparsity results in
efficiency while well connectedness can preserve the expressive power of the
CNNs. We use a well-studied class of graphs from theoretical computer science
that satisfies these properties known as Expander graphs. Expander graphs are
used to model connections between filters in CNNs to design networks called
X-Nets. We present two guarantees on the connectivity of X-Nets: Each node
influences every node in a layer in logarithmic steps, and the number of paths
between two sets of nodes is proportional to the product of their sizes. We
also propose efficient training and inference algorithms, making it possible to
train deeper and wider X-Nets effectively.
Expander based models give a 4% improvement in accuracy on MobileNet over
grouped convolutions, a popular technique, which has the same sparsity but
worse connectivity. X-Nets give better performance trade-offs than the original
ResNet and DenseNet-BC architectures. We achieve model sizes comparable to
state-of-the-art pruning techniques using our simple architecture design,
without any pruning. We hope that this work motivates other approaches to
utilize results from graph theory to develop efficient network architectures.

本文提出了一种图形连接的 CNN 结构，其中使用扩展器图以提高连接效率和层间信息传递，与现有的 ResNets 和 DenseNet 及其变体相比可以在精度和效率方面达到更好的平衡。