Semantic segmentation has witnessed remarkable advancements with the
adaptation of the Transformer architecture. Parallel to the strides made by the
Transformer, CNN-based U-Net has seen significant progress, especially in
high-resolution medical imaging and remote sensing. This dual success inspired
us to merge the strengths of both, leading to the inception of a U-Net-based
vision transformer decoder tailored for efficient contextual encoding. Here, we
propose a novel transformer decoder, U-MixFormer, built upon the U-Net
structure, designed for efficient semantic segmentation. Our approach
distinguishes itself from the previous transformer methods by leveraging
lateral connections between the encoder and decoder stages as feature queries
for the attention modules, apart from the traditional reliance on skip
connections. Moreover, we innovatively mix hierarchical feature maps from
various encoder and decoder stages to form a unified representation for keys
and values, giving rise to our unique mix-attention module. Our approach
demonstrates state-of-the-art performance across various configurations.
Extensive experiments show that U-MixFormer outperforms SegFormer, FeedFormer,
and SegNeXt by a large margin. For example, U-MixFormer-B0 surpasses
SegFormer-B0 and FeedFormer-B0 with 3.8% and 2.0% higher mIoU and 27.3% and
21.8% less computation and outperforms SegNext with 3.3% higher mIoU with
MSCAN-T encoder on ADE20K. Code available at
this https URL

我们提出了一种新的变换器解码器 U-MixFormer，基于 U-Net 结构设计的，用于高效的语义分割。通过在编码器和解码器阶段之间利用侧连接作为特征查询，我们的方法与以前的变换器方法有所不同。此外，我们创新地混合来自各个编码器和解码器阶段的分层特征图，形成一个统一的键和值表示，从而产生我们独特的混合注意模块。大量实验证明，U-MixFormer 在各种配置上表现出色，并且在 ADE20K 上使用 MSCAN-T 编码器的 mIoU 比 SegFormer 和 FeedFormer 高出 3.3%。