Open-vocabulary semantic segmentation is a challenging task, which requires
the model to output semantic masks of an image beyond a close-set vocabulary.
Although many efforts have been made to utilize powerful CLIP models to
accomplish this task, they are still easily overfitting to training classes due
to the natural gaps in semantic information between training and new classes.
To overcome this challenge, we propose a novel framework for openvocabulary
semantic segmentation called EBSeg, incorporating an Adaptively Balanced
Decoder (AdaB Decoder) and a Semantic Structure Consistency loss (SSC Loss).
The AdaB Decoder is designed to generate different image embeddings for both
training and new classes. Subsequently, these two types of embeddings are
adaptively balanced to fully exploit their ability to recognize training
classes and generalization ability for new classes. To learn a consistent
semantic structure from CLIP, the SSC Loss aligns the inter-classes affinity in
the image feature space with that in the text feature space of CLIP, thereby
improving the generalization ability of our model. Furthermore, we employ a
frozen SAM image encoder to complement the spatial information that CLIP
features lack due to the low training image resolution and image-level
supervision inherent in CLIP. Extensive experiments conducted across various
benchmarks demonstrate that the proposed EBSeg outperforms the state-of-the-art
methods. Our code and trained models will be here:
this https URL

提出了一种名为 EBSeg 的开放词汇语义分割框架，利用自适应平衡解码器和语义结构一致性损失来解决模型在训练和新类别之间存在的语义信息差异问题。通过生成不同的图像嵌入表征用于训练和新类别以及优化基于 CLIP 的模型的泛化能力，该方法在各种基准测试中表现出超越其他方法的性能。