We propose Clustering Mask Transformer (CMT-DeepLab), a transformer-based framework for panoptic segmentation designed around clustering. It rethinks the existing transformer architectures used in segmentation and detection; CMT-DeepLab considers the object queries as cluster centers, which fill the role of grouping the pixels when applied to segmentation. The clustering is computed with an alternating procedure, by first assigning pixels to the clusters by their feature affinity, and then updating the cluster centers and pixel features. Together, these operations comprise the Clustering Mask Transformer (CMT) layer, which produces cross-attention that is denser and more consistent with the final segmentation task. CMT-DeepLab improves the performance over prior art significantly by 4.4% PQ, achieving a new state-of-the-art of 55.7% PQ on the COCO test-dev set.

本文提出基于聚类的分割框架 Clustering Mask Transformer (CMT-DeepLab)，该框架重构了现有的用于分割和检测的 Transformer 架构；CMT-DeepLab 将对象查询视为聚类中心，这些聚类中心可用于分割中像素的分组，并通过更密集、更一致的交叉注意力实现更好的分割结果。实验结果表明，CMT-DeepLab 在 COCO 测试集上达到了 55.7% 的 PQ，并较先前方法提高了 4.4% 的 PQ。

CMT-DeepLab：用于全景分割的聚类蒙版Transformer