Fully supervised semantic segmentation learns from dense masks, which requires heavy annotation cost for closed set. In this paper, we use natural language as supervision without any pixel-level annotation for open world segmentation. We call the proposed framework as FreeSeg, where the mask is freely available from raw feature map of pretraining model. Compared with zero-shot or openset segmentation, FreeSeg doesn't require any annotated masks, and it widely predicts categories beyond class-agnostic unsupervised segmentation. Specifically, FreeSeg obtains free mask from Image-Text Similarity Map (ITSM) of Interpretable Contrastive Language-Image Pretraining (ICLIP). And our core improvements are the smoothed min pooling for dense ICLIP, with the partial label and pixel strategies for segmentation. Furthermore, FreeSeg is very straight forward without complex design like grouping, clustering or retrieval. Besides the simplicity, the performances of FreeSeg surpass previous state-of-the-art at large margins, e.g. 13.4% higher at mIoU on VOC dataset in the same settings.

本文提出了一种基于自然语言监督和预训练模型的FreeSeg框架，用于开放世界分割任务。它通过Image-Text Similarity Map（ITSM）获得自由掩码以进行语义分割，而不需要像传统方式一样像素级别的标注。该方法采用了平滑最小池化，局部标签和像素策略等关键技术，使性能超越当前先进水平。

FreeSeg：基于可解释对比语言-图像预训练的自由语义分割掩码