Model compression techniques on Deep Neural Network (DNN) have been widely
acknowledged as an effective way to achieve acceleration on a variety of
platforms, and DNN weight pruning is a straightforward and effective method.
There are currently two mainstreams of pruning methods representing two
extremes of pruning regularity: non-structured, fine-grained pruning can
achieve high sparsity and accuracy, but is not hardware friendly; structured,
coarse-grained pruning exploits hardware-efficient structures in pruning, but
suffers from accuracy drop when the pruning rate is high. In this paper, we
introduce PCONV, comprising a new sparsity dimension, -- fine-grained pruning
patterns inside the coarse-grained structures. PCONV comprises two types of
sparsities, Sparse Convolution Patterns (SCP) which is generated from
intra-convolution kernel pruning and connectivity sparsity generated from
inter-convolution kernel pruning. Essentially, SCP enhances accuracy due to its
special vision properties, and connectivity sparsity increases pruning rate
while maintaining balanced workload on filter computation. To deploy PCONV, we
develop a novel compiler-assisted DNN inference framework and execute PCONV
models in real-time without accuracy compromise, which cannot be achieved in
prior work. Our experimental results show that, PCONV outperforms three
state-of-art end-to-end DNN frameworks, TensorFlow-Lite, TVM, and Alibaba
Mobile Neural Network with speedup up to 39.2x, 11.4x, and 6.3x, respectively,
with no accuracy loss. Mobile devices can achieve real-time inference on
large-scale DNNs.

介绍了新的稀疏度维度 - 粗粒度结构内的细粒度剪枝模式 (PCONV) 方法，通过结构内的细粒度稀疏性和连接性稀疏性提高了精度和计算效率，在实时推理中取得了显著的速度提升，没有准确性损失。