With the fast growth of parameter size, it becomes increasingly challenging
to deploy large generative models as they typically require large GPU memory
consumption and massive computation. Unstructured model pruning has been a
common approach to reduce both GPU memory footprint and the overall computation
while retaining good model accuracy. However, the existing solutions do not
provide a highly-efficient support for handling unstructured sparsity on modern
GPUs, especially on the highly-structured Tensor Core hardware. Therefore, we
propose Flash-LLM for enabling low-cost and highly-efficient large generative
model inference with the sophisticated support of unstructured sparsity on
high-performance but highly restrictive Tensor Cores. Based on our key
observation that the main bottleneck of generative model inference is the
several skinny matrix multiplications for which Tensor Cores would be
significantly under-utilized due to low computational intensity, we propose a
general Load-as-Sparse and Compute-as-Dense methodology for unstructured sparse
matrix multiplication. The basic insight is to address the significant memory
bandwidth bottleneck while tolerating redundant computations that are not
critical for end-to-end performance on Tensor Cores. Based on this, we design
an effective software framework for Tensor Core based unstructured SpMM,
leveraging on-chip resources for efficient sparse data extraction and
computation/memory-access overlapping. At SpMM kernel level, Flash-LLM
significantly outperforms the state-of-the-art library, i.e., Sputnik and
SparTA by an average of 2.9x and 1.5x, respectively. At end-to-end framework
level on OPT-30B/66B/175B models, for tokens per GPU-second, Flash-LLM achieves
up to 3.8x and 3.6x improvement over DeepSpeed and FasterTransformer,
respectively, with significantly lower inference cost.

Flash-LLM 是一种针对大型生成模型的低成本高效大规模推断框架，通过优化稀疏矩阵乘法，在高性能 Tensor Cores 上实现了显著的性能提升。

Flash-LLM：使用非结构稀疏性实现成本效益高且高效的大型生成模型推断

Flash-LLM: Enabling Cost-Effective and Highly-Efficient Large Generative  Model Inference with Unstructured Sparsity

This paper presents a Spatial Re-parameterization (SpRe) method for the N:M
sparsity in CNNs. SpRe is stemmed from an observation regarding the restricted
variety in spatial sparsity present in N:M sparsity compared with unstructured
sparsity. Particularly, N:M sparsity exhibits a fixed sparsity rate within the
spatial domains due to its distinctive pattern that mandates N non-zero
components among M successive weights in the input channel dimension of
convolution filters. On the contrary, we observe that unstructured sparsity
displays a substantial divergence in sparsity across the spatial domains, which
we experimentally verified to be very crucial for its robust performance
retention compared with N:M sparsity. Therefore, SpRe employs the
spatial-sparsity distribution of unstructured sparsity to assign an extra
branch in conjunction with the original N:M branch at training time, which
allows the N:M sparse network to sustain a similar distribution of spatial
sparsity with unstructured sparsity. During inference, the extra branch can be
further re-parameterized into the main N:M branch, without exerting any
distortion on the sparse pattern or additional computation costs. SpRe has
achieved a commendable feat by matching the performance of N:M sparsity methods
with state-of-the-art unstructured sparsity methods across various benchmarks.
Code and models are anonymously available at
https://github.com/zyxxmu/SpRe.

本文提出了 Spatial Re-parameterization (SpRe) 方法，用于 N:M 稀疏性在 CNN 中的应用。该方法可以在训练期间利用非结构化稀疏性的空间稀疏性分布来为 N:M 稀疏性模型分配一个额外的支路，使其可以维持类似于非结构化稀疏性的空间稀疏性分布。在推理期间，该额外的支路可以进一步重新参数化为主要的 N:M 支路，而不会对稀疏模式或额外的计算成本产生扭曲。SpRe 方法可以在各种基准测试中匹配 N:M 稀疏性方法的表现与最先进的非结构化稀疏性方法。

N:M 稀疏性的空间重参数化

Spatial Re-parameterization for N:M Sparsity

Sparsity in Deep Neural Networks (DNNs) is studied extensively with the focus
of maximizing prediction accuracy given an overall parameter budget. Existing
methods rely on uniform or heuristic non-uniform sparsity budgets which have
sub-optimal layer-wise parameter allocation resulting in a) lower prediction
accuracy or b) higher inference cost (FLOPs). This work proposes Soft Threshold
Reparameterization (STR), a novel use of the soft-threshold operator on DNN
weights. STR smoothly induces sparsity while learning pruning thresholds
thereby obtaining a non-uniform sparsity budget. Our method achieves
state-of-the-art accuracy for unstructured sparsity in CNNs (ResNet50 and
MobileNetV1 on ImageNet-1K), and, additionally, learns non-uniform budgets that
empirically reduce the FLOPs by up to 50%. Notably, STR boosts the accuracy
over existing results by up to 10% in the ultra sparse (99%) regime and can
also be used to induce low-rank (structured sparsity) in RNNs. In short, STR is
a simple mechanism which learns effective sparsity budgets that contrast with
popular heuristics. Code, pretrained models and sparsity budgets are at
this https URL

本研究提出了一种基于软阈值重参数化（STR）的方法，该方法通过学习修剪阈值实现平滑稀疏（非均匀稀疏），从而获得非均匀稀疏预算，进而在 CNNs 中实现非结构化稀疏，在 ImageNet-1K 上实现了最先进的准确性，并且可以用于在 RNNs 中引出低秩（结构稀疏）。