Contemporary cutting-edge open-vocabulary segmentation approaches commonly rely on image-mask-text triplets, yet this restricted annotation is labour-intensive and encounters scalability hurdles in complex real-world scenarios. Although some methods are proposed to reduce the annotation cost with only text supervision, the incompleteness of supervision severely limits the versatility and performance. In this paper, we liberate the strict correspondence between masks and texts by using independent image-mask and image-text pairs, which can be easily collected respectively. With this unpaired mask-text supervision, we propose a new weakly-supervised open-vocabulary segmentation framework (Uni-OVSeg) that leverages confident pairs of mask predictions and entities in text descriptions. Using the independent image-mask and image-text pairs, we predict a set of binary masks and associate them with entities by resorting to the CLIP embedding space. However, the inherent noise in the correspondence between masks and entities poses a significant challenge when obtaining reliable pairs. In light of this, we advocate using the large vision-language model (LVLM) to refine text descriptions and devise a multi-scale ensemble to stablise the matching between masks and entities. Compared to text-only weakly-supervised methods, our Uni-OVSeg achieves substantial improvements of 15.5% mIoU on the ADE20K datasets, and even surpasses fully-supervised methods on the challenging PASCAL Context-459 dataset.

使用独立的图像-蒙版和图像-文本对，利用不配对的蒙版-文本监督，提出了一种新的弱监督开放词汇分割框架 (Uni-OVSeg)，通过利用自信的蒙版预测和文本描述中的实体，在CLIP嵌入空间中关联一组二进制蒙版和实体，采用大型视觉-语言模型 (LVLM) 对文本进行精炼，设计多尺度集成以稳定蒙版和实体之间的匹配，相较于仅使用文本进行弱监督的方法，在ADE20K数据集上实现了15.5% mIoU的显著改进，在具有挑战性的PASCAL Context-459数据集上甚至超过了完全监督的方法。

无配对掩码-文本监督下的开放词汇分割