Due to the depth degradation effect in residual connections, many efficient
Vision Transformers models that rely on stacking layers for information
exchange often fail to form sufficient information mixing, leading to unnatural
visual perception. To address this issue, in this paper, we propose Aggregated
Attention, a biomimetic design-based token mixer that simulates biological
foveal vision and continuous eye movement while enabling each token on the
feature map to have a global perception. Furthermore, we incorporate learnable
tokens that interact with conventional queries and keys, which further
diversifies the generation of affinity matrices beyond merely relying on the
similarity between queries and keys. Our approach does not rely on stacking for
information exchange, thus effectively avoiding depth degradation and achieving
natural visual perception. Additionally, we propose Convolutional GLU, a
channel mixer that bridges the gap between GLU and SE mechanism, which empowers
each token to have channel attention based on its nearest neighbor image
features, enhancing local modeling capability and model robustness. We combine
aggregated attention and convolutional GLU to create a new visual backbone
called TransNeXt. Extensive experiments demonstrate that our TransNeXt achieves
state-of-the-art performance across multiple model sizes. At a resolution of
$224^2$, TransNeXt-Tiny attains an ImageNet accuracy of 84.0%, surpassing
ConvNeXt-B with 69% fewer parameters. Our TransNeXt-Base achieves an ImageNet
accuracy of 86.2% and an ImageNet-A accuracy of 61.6% at a resolution of
$384^2$, a COCO object detection mAP of 57.1, and an ADE20K semantic
segmentation mIoU of 54.7.

通过仿生设计的令牌混合器，模拟生物视觉和持续眼动，提出聚合注意力机制（Aggregated Attention），使特征图上的每个令牌具有全局感知能力。同时，结合学习性令牌、卷积 GLU 机制，构建新的视觉主干网络 TransNeXt，实验结果表明其在多个模型尺寸上实现了最先进的性能。