Convolution neural networks (CNNs) have achieved remarkable success, but
typically accompany high computation cost and numerous redundant weight
parameters. To reduce the FLOPs, structure pruning is a popular approach to
remove the entire hidden structures via introducing coarse-grained sparsity.
Meanwhile, plentiful pruning works leverage fine-grained sparsity instead
(sparsity are randomly distributed), whereas their sparse models lack special
designed computing library for potential speedup. In this technical report, we
study and present an efficient convolution neural network inference system to
accelerate its forward pass by utilizing the fine-grained sparsity of
compressed CNNs. Our developed FSCNN is established based on a set of
specialized designed sparse data structures, operators and associated
algorithms. Experimentally, we validate that FSCNN outperforms standard deep
learning library PyTorch on popular CNN architectures such as VGG16 if
sufficiently high sparsity exhibits. However, due to the contiguity issue of
sparse operators, FSCNN is typically not comparable with highly optimized dense
operator. Therefore, coarse-grained (structured) sparsity is our recommendation
for generic model compression.

本文主要介绍了利用大规模稀疏结构提高卷积神经网络推理效率的方法， 开发了一种名为 FSCNN 的稀疏卷积神经网络推理系统，能够在一定的稀疏程度下优于标准深度学习库 PyTorch，但遇到高优化密集运算时推理效率有一定局限性，因此推荐采用粗粒度稀疏度进行通用模型压缩。