Open-vocabulary semantic segmentation models aim to accurately assign a
semantic label to each pixel in an image from a set of arbitrary
open-vocabulary texts. In order to learn such pixel-level alignment, current
approaches typically rely on a combination of (i) image-level VL model (e.g.
CLIP), (ii) ground truth masks, and (iii) custom grouping encoders. In this
paper, we introduce S-Seg, a novel model that can achieve surprisingly strong
performance without depending on any of the above elements. S-Seg leverages
pseudo-mask and language to train a MaskFormer, and can be easily trained from
publicly available image-text datasets. Contrary to prior works, our model
directly trains for pixel-level features and language alignment. Once trained,
S-Seg generalizes well to multiple testing datasets without requiring
fine-tuning. In addition, S-Seg has the extra benefits of scalability with data
and consistently improvement when augmented with self-training. We believe that
our simple yet effective approach will serve as a solid baseline for future
research.

通过 S-Seg 模型，我们可以实现准确的像素级别标签分配，无需依赖于图像级别的 VL 模型、地面实况掩码和自定义分组编码器，并且可以在多个测试数据集上进行良好的泛化而无需进行微调。