Although vision transformers (ViTs) have achieved great success in computer
vision, the heavy computational cost hampers their applications to dense
prediction tasks such as semantic segmentation on mobile devices. In this
paper, we present a mobile-friendly architecture named \textbf{To}ken
\textbf{P}yramid Vision Trans\textbf{former} (\textbf{TopFormer}). The proposed
\textbf{TopFormer} takes Tokens from various scales as input to produce
scale-aware semantic features, which are then injected into the corresponding
tokens to augment the representation. Experimental results demonstrate that our
method significantly outperforms CNN- and ViT-based networks across several
semantic segmentation datasets and achieves a good trade-off between accuracy
and latency. On the ADE20K dataset, TopFormer achieves 5\% higher accuracy in
mIoU than MobileNetV3 with lower latency on an ARM-based mobile device.
Furthermore, the tiny version of TopFormer achieves real-time inference on an
ARM-based mobile device with competitive results. The code and models are
available at: this https URL

本研究提出了一种名为 TopFormer 的移动友好的架构，该架构以不同尺度的 Token 作为输入，产生具有尺度感知的语义特征，并将这些特征注入到相应的 Token 中以增强表示能力，该方法在多个语义分割数据集上明显优于 CNN 和 ViT 网络，并在 ARM 移动设备上以较低的延迟实现了比 MobileNetV3 更高的平均交并比精度，同时，TopFormer 的微型版本可以在 ARM 移动设备上实现实时推断。