Vision Transformers (ViTs) have emerged with superior performance on computer
vision tasks compared to convolutional neural network (CNN)-based models.
However, ViTs are mainly designed for image classification that generate
single-scale low-resolution representations, which makes dense prediction tasks
such as semantic segmentation challenging for ViTs. Therefore, we propose
HRViT, which enhances ViTs to learn semantically-rich and spatially-precise
multi-scale representations by integrating high-resolution multi-branch
architectures with ViTs. We balance the model performance and efficiency of
HRViT by various branch-block co-optimization techniques. Specifically, we
explore heterogeneous branch designs, reduce the redundancy in linear layers,
and augment the attention block with enhanced expressiveness. Those approaches
enabled HRViT to push the Pareto frontier of performance and efficiency on
semantic segmentation to a new level, as our evaluation results on ADE20K and
Cityscapes show. HRViT achieves 50.20% mIoU on ADE20K and 83.16% mIoU on
Cityscapes, surpassing state-of-the-art MiT and CSWin backbones with an average
of +1.78 mIoU improvement, 28% parameter saving, and 21% FLOPs reduction,
demonstrating the potential of HRViT as a strong vision backbone for semantic
segmentation.

HRViT 通过将高分辨率多支路结构与 Vision Transformers 相集成，探索异构分支设计，减少线性层中的冗余并增强注意力块的表现力，从而在 ADE20K 和 Cityscapes 数据集达到 50.20％和 83.16％的 mIoU 优化性能和效率，比现有的 MiT 和 CSWin 骨干框架平均提高 1.78 个 mIoU，参数节省 28％，FLOPs 减少 21％。