In recent years, Transformer-based language models have become the standard
approach for natural language processing tasks. However, stringent throughput
and latency requirements in industrial applications are limiting their
adoption. To mitigate the gap, model compression techniques such as structured
pruning are being used to improve inference efficiency. However, most existing
neural network inference runtimes lack adequate support for structured
sparsity. In this paper, we propose an efficient sparse deep learning inference
software stack for Transformer-based language models where the weights are
pruned with constant block size. Our sparse software accelerator leverages
Intel Deep Learning Boost to maximize the performance of sparse matrix - dense
matrix multiplication (commonly abbreviated as SpMM) on CPUs. Our SpMM kernel
outperforms the existing sparse libraries (oneMKL, TVM, and LIBXSMM) by an
order of magnitude on a wide range of GEMM shapes under 5 representative
sparsity ratios (70%, 75%, 80%, 85%, 90%). Moreover, our SpMM kernel shows up
to 5x speedup over dense GEMM kernel of oneDNN, a well-optimized dense library
widely used in industry. We apply our sparse accelerator on widely-used
Transformer-based language models including Bert-Mini, DistilBERT, Bert-Base,
and BERT-Large. Our sparse inference software shows up to 1.5x speedup over
Neural Magic's Deepsparse under same configurations on Xeon on Amazon Web
Services under proxy production latency constraints. We also compare our
solution with two framework-based inference solutions, ONNX Runtime and
PyTorch, and demonstrate up to 37x speedup over ONNX Runtime and 345x over
PyTorch on Xeon under the latency constraints. All the source code is publicly
available on Github: this https URL

本论文提出了一种基于定常块大小的高效稀疏深度学习推断软件堆栈，可以将稀疏加速器应用于 Transformer-based 语言模型，我们的稀疏加速器在处理各种 GEMM 形状时，比现有的稀疏库快一个数量级，在 Xeon 上具有高效的性能。