Convolution neural networks (CNNs) have achieved remarkable success, but
typically accompany high computation cost and numerous redundant weight
parameters. To reduce the FLOPs, structure pruning is a popular approach to
remove the entire hidden structures via introducing coarse-grained sparsity.
Meanwhile, plentiful pruning works leverage fine-grained sparsity instead
(sparsity are randomly distributed), whereas their sparse models lack special
designed computing library for potential speedup. In this technical report, we
study and present an efficient convolution neural network inference system to
accelerate its forward pass by utilizing the fine-grained sparsity of
compressed CNNs. Our developed FSCNN is established based on a set of
specialized designed sparse data structures, operators and associated
algorithms. Experimentally, we validate that FSCNN outperforms standard deep
learning library PyTorch on popular CNN architectures such as VGG16 if
sufficiently high sparsity exhibits. However, due to the contiguity issue of
sparse operators, FSCNN is typically not comparable with highly optimized dense
operator. Therefore, coarse-grained (structured) sparsity is our recommendation
for generic model compression.

本文主要介绍了利用大规模稀疏结构提高卷积神经网络推理效率的方法， 开发了一种名为 FSCNN 的稀疏卷积神经网络推理系统，能够在一定的稀疏程度下优于标准深度学习库 PyTorch，但遇到高优化密集运算时推理效率有一定局限性，因此推荐采用粗粒度稀疏度进行通用模型压缩。

FSCNN: 一种快速的稀疏卷积神经网络推理系统

FSCNN: A Fast Sparse Convolution Neural Network Inference System

In trained deep neural networks, unstructured pruning can reduce redundant
weights to lower storage cost. However, it requires the customization of
hardwares to speed up practical inference. Another trend accelerates sparse
model inference on general-purpose hardwares by adopting coarse-grained
sparsity to prune or regularize consecutive weights for efficient computation.
But this method often sacrifices model accuracy. In this paper, we propose a
novel fine-grained sparsity approach, balanced sparsity, to achieve high model
accuracy with commercial hardwares efficiently. Our approach adapts to high
parallelism property of GPU, showing incredible potential for sparsity in the
widely deployment of deep learning services. Experiment results show that
balanced sparsity achieves up to 3.1x practical speedup for model inference on
GPU, while retains the same high model accuracy as fine-grained sparsity.

本文提出一种新的 fine-grained sparsity 方法，即平衡稀疏化方法，使得深度学习模型在商用硬件上得以高效地获得准确性和加速，同时利用 GPU 的高并行特性，取得了高达 3.1 倍的实际加速效果。