Text-to-image diffusion techniques have shown exceptional capability of
producing high-quality images from text descriptions. This indicates that there
exists a strong correlation between the visual and textual domains. In
addition, text-image discriminative models such as CLIP excel in image
labelling from text prompts, thanks to the rich and diverse information
available from open concepts. In this paper, we leverage these technical
advances to solve a challenging problem in computer vision: camouflaged
instance segmentation. Specifically, we propose a method built upon a
state-of-the-art diffusion model, empowered by open-vocabulary to learn
multi-scale textual-visual features for camouflaged object representations.
Such cross-domain representations are desirable in segmenting camouflaged
objects where visual cues are subtle to distinguish the objects from the
background, especially in segmenting novel objects which are not seen in
training. We also develop technically supportive components to effectively fuse
cross-domain features and engage relevant features towards respective
foreground objects. We validate our method and compare it with existing ones on
several benchmark datasets of camouflaged instance segmentation and generic
open-vocabulary instance segmentation. Experimental results confirm the
advances of our method over existing ones. We will publish our code and
pre-trained models to support future research.

通过结合文本到图像扩散技术和开放词汇，我们提出了一种解决计算机视觉中伪装实例分割问题的方法，该方法利用跨领域的文本 - 视觉特征学习来区分复杂背景中伪装对象，实验证明了该方法相较于现有方法的优势。

利用开放词汇扩散进行伪装实例分割

Leveraging Open-Vocabulary Diffusion to Camouflaged Instance  Segmentation

We introduce Open3DIS, a novel solution designed to tackle the problem of
Open-Vocabulary Instance Segmentation within 3D scenes. Objects within 3D
environments exhibit diverse shapes, scales, and colors, making precise
instance-level identification a challenging task. Recent advancements in
Open-Vocabulary scene understanding have made significant strides in this area
by employing class-agnostic 3D instance proposal networks for object
localization and learning queryable features for each 3D mask. While these
methods produce high-quality instance proposals, they struggle with identifying
small-scale and geometrically ambiguous objects. The key idea of our method is
a new module that aggregates 2D instance masks across frames and maps them to
geometrically coherent point cloud regions as high-quality object proposals
addressing the above limitations. These are then combined with 3D
class-agnostic instance proposals to include a wide range of objects in the
real world. To validate our approach, we conducted experiments on three
prominent datasets, including ScanNet200, S3DIS, and Replica, demonstrating
significant performance gains in segmenting objects with diverse categories
over the state-of-the-art approaches.

Open3DIS 是对 3D 场景中开放词汇实例分割问题的一种新解决方案，通过聚合帧间的 2D 实例掩码并将其映射到几何连贯的点云区域作为高质量的对象提案，与 3D 非特定类实例提案相结合，能够在真实世界的广泛对象中实现显著的性能提升。

Open3DIS: 基于 2D 掩码引导的开放词汇 3D 实例分割

Open3DIS: Open-vocabulary 3D Instance Segmentation with 2D Mask Guidance

Open-vocabulary instance segmentation aims at segmenting novel classes
without mask annotations. It is an important step toward reducing laborious
human supervision. Most existing works first pretrain a model on captioned
images covering many novel classes and then finetune it on limited base classes
with mask annotations. However, the high-level textual information learned from
caption pretraining alone cannot effectively encode the details required for
pixel-wise segmentation. To address this, we propose a cross-modal
pseudo-labeling framework, which generates training pseudo masks by aligning
word semantics in captions with visual features of object masks in images.
Thus, our framework is capable of labeling novel classes in captions via their
word semantics to self-train a student model. To account for noises in pseudo
masks, we design a robust student model that selectively distills mask
knowledge by estimating the mask noise levels, hence mitigating the adverse
impact of noisy pseudo masks. By extensive experiments, we show the
effectiveness of our framework, where we significantly improve mAP score by
4.5% on MS-COCO and 5.1% on the large-scale Open Images & Conceptual Captions
datasets compared to the state-of-the-art.

本文提出一种跨模态伪标签（cross-modal pseudo-labeling）框架，用于面向开放词汇的实例分割，通过与对象掩膜的视觉特征对齐，实现对字词语义中的新类别进行标记，从而自我训练出一个学生模型，缓解了伪掩膜中存在的噪声干扰所带来的不良影响，相比现有工作，在 MS-COCO 和 Open Images & Conceptual Captions 数据集上分别实现了 4.5% 和 5.1% 的 mAP 得分提高。