The recently proposed MaskFormer gives a refreshed perspective on the task of
semantic segmentation: it shifts from the popular pixel-level classification
paradigm to a mask-level classification method. In essence, it generates paired
probabilities and masks corresponding to category segments and combines them
during inference for the segmentation maps. In our study, we find that per-mask
classification decoder on top of a single-scale feature is not effective enough
to extract reliable probability or mask. To mine for rich semantic information
across the feature pyramid, we propose a transformer-based Pyramid Fusion
Transformer (PFT) for per-mask approach semantic segmentation with multi-scale
features. The proposed transformer decoder performs cross-attention between the
learnable queries and each spatial feature from the feature pyramid in parallel
and uses cross-scale inter-query attention to exchange complimentary
information. We achieve competitive performance on three widely used semantic
segmentation datasets. In particular, on ADE20K validation set, our result with
Swin-B backbone surpasses that of MaskFormer's with a much larger Swin-L
backbone in both single-scale and multi-scale inference, achieving 54.1 mIoU
and 55.7 mIoU respectively. Using a Swin-L backbone, we achieve single-scale
56.1 mIoU and multi-scale 57.4 mIoU, obtaining state-of-the-art performance on
the dataset. Extensive experiments on three widely used semantic segmentation
datasets verify the effectiveness of our proposed method.

本研究提出了一种基于 Transformer 的 Feature Pyramid Fusion Transformer (PFT) 方法，结合学习到的查询和 Feature Pyramid 中的每个空间特征进行交叉注意力和交叉尺度的互补信息交换，提高了面向掩码的语义分割性能，并在三个广泛使用的语义分割数据集上达到了与现有最佳方法相比的最先进的性能。