To address the challenge of increasing network size, researchers have
developed sparse models through network pruning. However, maintaining model
accuracy while achieving significant speedups on general computing devices
remains an open problem. In this paper, we present a novel mobile inference
acceleration framework SparseByteNN, which leverages fine-grained kernel
sparsity to achieve real-time execution as well as high accuracy. Our framework
consists of two parts: (a) A fine-grained kernel sparsity schema with a
sparsity granularity between structured pruning and unstructured pruning. It
designs multiple sparse patterns for different operators. Combined with our
proposed whole network rearrangement strategy, the schema achieves a high
compression rate and high precision at the same time. (b) Inference engine
co-optimized with the sparse pattern. The conventional wisdom is that this
reduction in theoretical FLOPs does not translate into real-world efficiency
gains. We aim to correct this misconception by introducing a family of
efficient sparse kernels for ARM and WebAssembly. Equipped with our efficient
implementation of sparse primitives, we show that sparse versions of
MobileNet-v1 outperform strong dense baselines on the efficiency-accuracy
curve. Experimental results on Qualcomm 855 show that for 30% sparse
MobileNet-v1, SparseByteNN achieves 1.27x speedup over the dense version and
1.29x speedup over the state-of-the-art sparse inference engine MNN with a
slight accuracy drop of 0.224%. The source code of SparseByteNN will be
available at this https URL

为了解决网络规模增大的挑战，研究者通过网络剪枝开发了稀疏模型。然而，在通用计算设备上实现显著加速的同时保持模型准确性仍然是一个未解决的问题。在本文中，我们提出了一种新颖的移动推理加速框架 SparseByteNN，通过利用细粒度的内核稀疏性实现实时执行和高准确性。我们的框架包括两个部分：（a）一种细粒度的内核稀疏模式，其稀疏粒度介于结构化剪枝和非结构化剪枝之间。它为不同操作器设计了多种稀疏模式。结合我们提出的整体网络重新排列策略，该模式同时实现高压缩率和高精度。（b）与稀疏模式共同优化的推理引擎。常理认为，这种理论上的 FLOPs 减少并不能转化为真实世界的效率提升。我们旨在通过引入一族针对 ARM 和 WebAssembly 的高效稀疏内核来纠正这种误解。通过我们对稀疏原语的高效实现，我们展示了 MobileNet-v1 的稀疏版本在效率 - 准确性曲线上胜过性能强大且密集的基线模型。Qualcomm 855 的实验结果表明，对于 30％稀疏的 MobileNet-v1，SparseByteNN 相对于密集版本获得了 1.27 倍的加速，并相对于最先进的稀疏推理引擎 MNN 获得了 1.29 倍的加速，准确性略有下降，下降了 0.224％。SparseByteNN 的源代码将在此 https URL 上提供。