The DETR-like segmentors have underpinned the most recent breakthroughs in
semantic segmentation, which end-to-end train a set of queries representing the
class prototypes or target segments. Recently, masked attention is proposed to
restrict each query to only attend to the foreground regions predicted by the
preceding decoder block for easier optimization. Although promising, it relies
on the learnable parameterized positional queries which tend to encode the
dataset statistics, leading to inaccurate localization for distinct individual
queries. In this paper, we propose a simple yet effective query design for
semantic segmentation termed Dynamic Focus-aware Positional Queries (DFPQ),
which dynamically generates positional queries conditioned on the
cross-attention scores from the preceding decoder block and the positional
encodings for the corresponding image features, simultaneously. Therefore, our
DFPQ preserves rich localization information for the target segments and
provides accurate and fine-grained positional priors. In addition, we propose
to efficiently deal with high-resolution cross-attention by only aggregating
the contextual tokens based on the low-resolution cross-attention scores to
perform local relation aggregation. Extensive experiments on ADE20K and
Cityscapes show that with the two modifications on Mask2former, our framework
achieves SOTA performance and outperforms Mask2former by clear margins of 1.1%,
1.9%, and 1.1% single-scale mIoU with ResNet-50, Swin-T, and Swin-B backbones
on the ADE20K validation set, respectively. Source code is available at
this https URL

本文提出了一种名为 DFPQ（Dynamic Focus-aware Positional Queries）的 query 设计方法，该方法基于前一个解码器块的跨注意力得分和相应图像特征的位置编码动态生成位置查询，同时通过仅基于低分辨率跨注意力分数聚合上下文令牌以执行局部关系聚合。经实验证明，在 ADE20K 和 Cityscapes 数据集上，该方法在 Mask2former 的基础上实现了 SOTA 表现，且 ResNet-50、Swin-T 和 Swin-B 等背骨在 ADE20K 验证集上的单尺度 mIoU 分别优于 Mask2former 1.1％，1.9％和 1.1％。

面向语义分割的动态焦点感知位置查询

Dynamic Focus-aware Positional Queries for Semantic Segmentation

Image segmentation is about grouping pixels with different semantics, e.g.,
category or instance membership, where each choice of semantics defines a task.
While only the semantics of each task differ, current research focuses on
designing specialized architectures for each task. We present Masked-attention
Mask Transformer (Mask2Former), a new architecture capable of addressing any
image segmentation task (panoptic, instance or semantic). Its key components
include masked attention, which extracts localized features by constraining
cross-attention within predicted mask regions. In addition to reducing the
research effort by at least three times, it outperforms the best specialized
architectures by a significant margin on four popular datasets. Most notably,
Mask2Former sets a new state-of-the-art for panoptic segmentation (57.8 PQ on
COCO), instance segmentation (50.1 AP on COCO) and semantic segmentation (57.7
mIoU on ADE20K).

提供一种新的架构 Mask2Former，该架构通过应用掩蔽注意力机制提取局部特征来解决多种类型的图像分割任务，包括全景分割、实体分割和语义分割，并且优于当前最佳的专门任务架构。