This paper presents CLUSTERFORMER, a universal vision model that is based on
the CLUSTERing paradigm with TransFORMER. It comprises two novel designs: 1.
recurrent cross-attention clustering, which reformulates the cross-attention
mechanism in Transformer and enables recursive updates of cluster centers to
facilitate strong representation learning; and 2. feature dispatching, which
uses the updated cluster centers to redistribute image features through
similarity-based metrics, resulting in a transparent pipeline. This elegant
design streamlines an explainable and transferable workflow, capable of
tackling heterogeneous vision tasks (i.e., image classification, object
detection, and image segmentation) with varying levels of clustering
granularity (i.e., image-, box-, and pixel-level). Empirical results
demonstrate that CLUSTERFORMER outperforms various well-known specialized
architectures, achieving 83.41% top-1 acc. over ImageNet-1K for image
classification, 54.2% and 47.0% mAP over MS COCO for object detection and
instance segmentation, 52.4% mIoU over ADE20K for semantic segmentation, and
55.8% PQ over COCO Panoptic for panoptic segmentation. For its efficacy, we
hope our work can catalyze a paradigm shift in universal models in computer
vision.

CLUSTERFORMER 是一种基于 CLUSTERING 范例和 TransFORMER 的通用视觉模型，通过递归更新聚类中心和基于相似度的指标来实现图像分类、目标检测和图像分割等多样化视觉任务的性能提升，并说明其在各项指标上优于其他专门化架构，为计算机视觉中的通用模型带来了范式转变。