The quadratic computational complexity to the number of tokens limits the
practical applications of Vision Transformers (ViTs). Several works propose to
prune redundant tokens to achieve efficient ViTs. However, these methods
generally suffer from (i) dramatic accuracy drops, (ii) application difficulty
in the local vision transformer, and (iii) non-general-purpose networks for
downstream tasks. In this work, we propose a novel Semantic Token ViT (STViT),
for efficient global and local vision transformers, which can also be revised
to serve as backbone for downstream tasks. The semantic tokens represent
cluster centers, and they are initialized by pooling image tokens in space and
recovered by attention, which can adaptively represent global or local semantic
information. Due to the cluster properties, a few semantic tokens can attain
the same effect as vast image tokens, for both global and local vision
transformers. For instance, only 16 semantic tokens on DeiT-(Tiny,Small,Base)
can achieve the same accuracy with more than 100% inference speed improvement
and nearly 60% FLOPs reduction; on Swin-(Tiny,Small,Base), we can employ 16
semantic tokens in each window to further speed it up by around 20% with slight
accuracy increase. Besides great success in image classification, we also
extend our method to video recognition. In addition, we design a
STViT-R(ecover) network to restore the detailed spatial information based on
the STViT, making it work for downstream tasks, which is powerless for previous
token sparsification methods. Experiments demonstrate that our method can
achieve competitive results compared to the original networks in object
detection and instance segmentation, with over 30% FLOPs reduction for
backbone. Code is available at this http URL

本文介绍了一种基于语义标记的 ViT 模型，可以用于图像分类以及物体检测和实例分割等任务，并通过对空间中的池化图像标记进行 attention 的方法，来取代大量的图像标记，从而实现了网络的降维升效。

从标记稀疏化视角提高视觉 Transformer 的效率

Making Vision Transformers Efficient from A Token Sparsification View

In this paper, we present a new approach for model acceleration by exploiting
spatial sparsity in visual data. We observe that the final prediction in vision
Transformers is only based on a subset of the most informative tokens, which is
sufficient for accurate image recognition. Based on this observation, we
propose a dynamic token sparsification framework to prune redundant tokens
progressively and dynamically based on the input to accelerate vision
Transformers. Specifically, we devise a lightweight prediction module to
estimate the importance score of each token given the current features. The
module is added to different layers to prune redundant tokens hierarchically.
While the framework is inspired by our observation of the sparse attention in
vision Transformers, we find the idea of adaptive and asymmetric computation
can be a general solution for accelerating various architectures. We extend our
method to hierarchical models including CNNs and hierarchical vision
Transformers as well as more complex dense prediction tasks that require
structured feature maps by formulating a more generic dynamic spatial
sparsification framework with progressive sparsification and asymmetric
computation for different spatial locations. By applying lightweight fast paths
to less informative features and using more expressive slow paths to more
important locations, we can maintain the structure of feature maps while
significantly reducing the overall computations. Extensive experiments
demonstrate the effectiveness of our framework on various modern architectures
and different visual recognition tasks. Our results clearly demonstrate that
dynamic spatial sparsification offers a new and more effective dimension for
model acceleration. Code is available at
this https URL

本文提出了一种通过利用视觉数据中的空间稀疏性进行模型加速的新方法，该方法基于所提出的动态令牌稀疏化框架，并通过自适应和不对称计算等方式推广到各种体系结构中，通过对不重要的特征使用轻量级快速路径和对更重要位置使用更具表现力的慢速路径，可以显著减少总体计算量，实验结果表明动态空间稀疏化为模型加速提供了新的更有效的解决方案。

动态空间稀疏化：用于高效视觉 Transformer 和卷积神经网络的优化

Dynamic Spatial Sparsification for Efficient Vision Transformers and Convolutional Neural Networks

Attention is sparse in vision transformers. We observe the final prediction
in vision transformers is only based on a subset of most informative tokens,
which is sufficient for accurate image recognition. Based on this observation,
we propose a dynamic token sparsification framework to prune redundant tokens
progressively and dynamically based on the input. Specifically, we devise a
lightweight prediction module to estimate the importance score of each token
given the current features. The module is added to different layers to prune
redundant tokens hierarchically. To optimize the prediction module in an
end-to-end manner, we propose an attention masking strategy to differentiably
prune a token by blocking its interactions with other tokens. Benefiting from
the nature of self-attention, the unstructured sparse tokens are still hardware
friendly, which makes our framework easy to achieve actual speed-up. By
hierarchically pruning 66% of the input tokens, our method greatly reduces
31%~37% FLOPs and improves the throughput by over 40% while the drop of
accuracy is within 0.5% for various vision transformers. Equipped with the
dynamic token sparsification framework, DynamicViT models can achieve very
competitive complexity/accuracy trade-offs compared to state-of-the-art CNNs
and vision transformers on ImageNet. Code is available at
this https URL

提出了一种动态的、基于输入的 Token 稀疏化框架，通过阈值删除多余的信息元素，并通过自注意力网络的特性，使其仍然具有硬件友好性。在保证精度的同时，相比现有 CNN 和 Transformer 模型，通过分层删减 66% 的输入元素，可以大大减少 FLOPs、提高吞吐量。