Recently, learning open-vocabulary semantic segmentation from text
supervision has achieved promising downstream performance. Nevertheless,
current approaches encounter an alignment granularity gap owing to the absence
of dense annotations, wherein they learn coarse image/region-text alignment
during training yet perform group/pixel-level predictions at inference. Such
discrepancy leads to suboptimal learning efficiency and inferior zero-shot
segmentation results. In this paper, we introduce a Multi-Grained Cross-modal
Alignment (MGCA) framework, which explicitly learns pixel-level alignment along
with object- and region-level alignment to bridge the granularity gap without
any dense annotations. Specifically, MGCA ingeniously constructs pseudo
multi-granular semantic correspondences upon image-text pairs and collaborates
with hard sampling strategies to facilitate fine-grained cross-modal
contrastive learning. Further, we point out the defects of existing group and
pixel prediction units in downstream segmentation and develop an adaptive
semantic unit which effectively mitigates their dilemmas including under- and
over-segmentation. Training solely on CC3M, our method achieves significant
advancements over state-of-the-art methods, demonstrating its effectiveness and
efficiency.

提出了一种多粒度跨模态对齐 (MGCA) 框架，通过在像素级、对象级和区域级学习对齐来解决现有方法在像素级预测和训练时的粒度差异问题，并采用硬采样策略促进精细的跨模态对比学习，进一步开发自适应语义单元来改善像素预测单元在下游分割中的缺陷。在 CC3M 数据集上进行训练后，该方法在性能上显著超过了现有的方法，验证了其有效性和高效性。