This paper studies co-segmenting the common semantic object in a set of
images. Existing works either rely on carefully engineered networks to mine the
implicit semantic information in visual features or require extra data (i.e.,
classification labels) for training. In this paper, we leverage the contrastive
language-image pre-training framework (CLIP) for the task. With a backbone
segmentation network that independently processes each image from the set, we
introduce semantics from CLIP into the backbone features, refining them in a
coarse-to-fine manner with three key modules: i) an image set feature
correspondence module, encoding global consistent semantic information of the
image set; ii) a CLIP interaction module, using CLIP-mined common semantics of
the image set to refine the backbone feature; iii) a CLIP regularization
module, drawing CLIP towards this co-segmentation task, identifying the best
CLIP semantic and using it to regularize the backbone feature. Experiments on
four standard co-segmentation benchmark datasets show that the performance of
our method outperforms state-of-the-art methods.

本文研究了在一组图像中共同语义对象的共同分割，并使用对比式语言 - 图像预训练框架 (CLIP) 来完成该任务。通过一个用于独立处理图像集中每个图像的骨干分割网络，我们引入了 CLIP 中的语义信息，并通过三个关键模块 ——i) 图像集特征对应模块，对图像集的全局一致语义信息进行编码；ii) CLIP 交互模块，利用 CLIP 提取的图像集的共同语义信息来优化骨干特征；iii) CLIP 正则化模块，引导 CLIP 朝向共同分割任务，找到最佳的 CLIP 语义并用其来正则化骨干特征 —— 以一种粗糙到精细的方式优化这些特征。在四个标准的共同分割基准数据集上的实验证明了我们方法的性能优于最先进的方法。

LCCo：Lending CLIP to Co-Segmentation

LCCo: Lending CLIP to Co-Segmentation

Semantic labeling for very high resolution (VHR) images in urban areas, is of
significant importance in a wide range of remote sensing applications. However,
many confusing manmade objects and intricate fine-structured objects make it
very difficult to obtain both coherent and accurate labeling results. For this
challenging task, we propose a novel deep model with convolutional neural
networks (CNNs), i.e., an end-to-end self-cascaded network (ScasNet).
Specifically, for confusing manmade objects, ScasNet improves the labeling
coherence with sequential global-to-local contexts aggregation. Technically,
multi-scale contexts are captured on the output of a CNN encoder, and then they
are successively aggregated in a self-cascaded manner. Meanwhile, for
fine-structured objects, ScasNet boosts the labeling accuracy with a
coarse-to-fine refinement strategy. It progressively refines the target objects
using the low-level features learned by CNN's shallow layers. In addition, to
correct the latent fitting residual caused by multi-feature fusion inside
ScasNet, a dedicated residual correction scheme is proposed. It greatly
improves the effectiveness of ScasNet. Extensive experimental results on three
public datasets, including two challenging benchmarks, show that ScasNet
achieves the state-of-the-art performance.

提出了一种基于 CNN 的新型深度模型（ScasNet），用于在具有高度复杂的城市环境下对非常高分辨率（VHR）图像进行语义标注。ScasNet 通过全局到局部环境聚合和粗到精微的细化策略，以及专用的残差校正方案，大大提高了标记的一致性和准确性，并在三个公共数据集上获得了最先进的性能。