Multi-head self-attention (MSA) is a key component of Vision Transformers
(ViTs), which have achieved great success in various vision tasks. However,
their high computational cost and memory footprint hinder their deployment on
resource-constrained devices. Conventional pruning approaches can only compress
and accelerate the MSA module using head pruning, although the head is not an
atomic unit. To address this issue, we propose a novel graph-aware neuron-level
pruning method, Structured Neuron-level Pruning (SNP). SNP prunes neurons with
less informative attention scores and eliminates redundancy among heads.
Specifically, it prunes graphically connected query and key layers having the
least informative attention scores while preserving the overall attention
scores. Value layers, which can be pruned independently, are pruned to
eliminate inter-head redundancy. Our proposed method effectively compresses and
accelerates Transformer-based models for both edge devices and server
processors. For instance, the DeiT-Small with SNP runs 3.1$\times$ faster than
the original model and achieves performance that is 21.94\% faster and 1.12\%
higher than the DeiT-Tiny. Additionally, SNP combine successfully with
conventional head or block pruning approaches. SNP with head pruning could
compress the DeiT-Base by 80\% of the parameters and computational costs and
achieve 3.85$\times$ faster inference speed on RTX3090 and 4.93$\times$ on
Jetson Nano.

我们提出了一种新颖的图感知神经元级剪枝方法（SNP），通过剪枝数据图中信息较少的注意力分数和消除多头冗余，有效压缩和加速基于 Transformer 的模型，适用于边缘设备和服务器处理器。