Zero-shot Panoptic Segmentation (ZPS) aims to recognize foreground instances
and background stuff without images containing unseen categories in training.
Due to the visual data sparsity and the difficulty of generalizing from seen to
unseen categories, this task remains challenging. To better generalize to
unseen classes, we propose Conditional tOken aligNment and Cycle trAnsiTion
(CONCAT), to produce generalizable semantic vision queries. First, a feature
extractor is trained by CON to link the vision and semantics for providing
target queries. Formally, CON is proposed to align the semantic queries with
the CLIP visual CLS token extracted from complete and masked images. To address
the lack of unseen categories, a generator is required. However, one of the
gaps in synthesizing pseudo vision queries, ie, vision queries for unseen
categories, is describing fine-grained visual details through semantic
embeddings. Therefore, we approach CAT to train the generator in
semantic-vision and vision-semantic manners. In semantic-vision, visual query
contrast is proposed to model the high granularity of vision by pulling the
pseudo vision queries with the corresponding targets containing segments while
pushing those without segments away. To ensure the generated queries retain
semantic information, in vision-semantic, the pseudo vision queries are mapped
back to semantic and supervised by real semantic embeddings. Experiments on ZPS
achieve a 5.2% hPQ increase surpassing SOTA. We also examine inductive ZPS and
open-vocabulary semantic segmentation and obtain comparative results while
being 2 times faster in testing.

为了更好地推广到看不见的类别，我们提出了条件令牌对齐和循环转换 (CONCAT)，以生成具有普适性的语义视觉查询，经过实验，在零样本全景分割 (ZPS) 上取得了 5.2% 的 hPQ 提高，并在测试中比 SOTA 快 2 倍。