The Vision Transformer (ViT) has gained prominence for its superior
relational modeling prowess. However, its global attention mechanism's
quadratic complexity poses substantial computational burdens. A common remedy
spatially groups tokens for self-attention, reducing computational
requirements. Nonetheless, this strategy neglects semantic information in
tokens, possibly scattering semantically-linked tokens across distinct groups,
thus compromising the efficacy of self-attention intended for modeling
inter-token dependencies. Motivated by these insights, we introduce a fast and
balanced clustering method, named \textbf{S}emantic \textbf{E}quitable
\textbf{C}lustering (SEC). SEC clusters tokens based on their global semantic
relevance in an efficient, straightforward manner. In contrast to traditional
clustering methods requiring multiple iterations, our method achieves token
clustering in a single pass. Additionally, SEC regulates the number of tokens
per cluster, ensuring a balanced distribution for effective parallel processing
on current computational platforms without necessitating further optimization.
Capitalizing on SEC, we propose a versatile vision backbone, SecViT.
Comprehensive experiments in image classification, object detection, instance
segmentation, and semantic segmentation validate to the effectiveness of
SecViT. Remarkably, SecViT attains an impressive \textbf{84.2\%} image
classification accuracy with only \textbf{27M} parameters and \textbf{4.4G}
FLOPs, without the need for for additional supervision or data. Code will be
available at https://github.com/qhfan/SecViT.

基于语义平等聚类（SEC）的全球注意力机制和视觉变换器（ViT）提出了一种速度快且平衡的聚类方法 SecViT，证明其在图像分类、目标检测、实例分割和语义分割等方面的有效性。