Despite the strong performance of Transformers, their quadratic computation
complexity presents challenges in applying them to vision tasks. Automatic
pruning is one of effective methods for reducing computation complexity without
heuristic approaches. However, directly applying it to multi-head attention is
not straightforward due to channel misalignment. In this paper, we propose an
automatic channel pruning method to take into account the multi-head attention
mechanism. First, we incorporate channel similarity-based weights into the
pruning indicator to preserve more informative channels in each head. Then, we
adjust pruning indicator to enforce removal of channels in equal proportions
across all heads, preventing the channel misalignment. We also add a reweight
module to compensate for information loss resulting from channel removal, and
an effective initialization step for pruning indicator based on difference of
attention between original structure and each channel. Our proposed method can
be used to not only original attention, but also linear attention, which is
more efficient as linear complexity with respect to the number of tokens. On
ImageNet-1K, applying our pruning method to the FLattenTransformer, which
includes both attention mechanisms, shows outperformed accuracy for several
MACs compared with previous state-of-the-art efficient models and pruned
methods. Code will be available soon.

基于通道相似性和修剪指示器的多头注意力机制自动修剪方法，通过平衡各头通道的移除比例和通道信息的重新加权来降低计算复杂性，并在图像分类任务中表现出超越先前的高效模型和修剪方法的准确性。