We introduce the task of open-vocabulary 3D instance segmentation.
Traditional approaches for 3D instance segmentation largely rely on existing 3D
annotated datasets, which are restricted to a closed-set of object categories.
This is an important limitation for real-life applications where one might need
to perform tasks guided by novel, open-vocabulary queries related to objects
from a wide variety. Recently, open-vocabulary 3D scene understanding methods
have emerged to address this problem by learning queryable features per each
point in the scene. While such a representation can be directly employed to
perform semantic segmentation, existing methods have limitations in their
ability to identify object instances. In this work, we address this limitation,
and propose OpenMask3D, which is a zero-shot approach for open-vocabulary 3D
instance segmentation. Guided by predicted class-agnostic 3D instance masks,
our model aggregates per-mask features via multi-view fusion of CLIP-based
image embeddings. We conduct experiments and ablation studies on the ScanNet200
dataset to evaluate the performance of OpenMask3D, and provide insights about
the open-vocabulary 3D instance segmentation task. We show that our approach
outperforms other open-vocabulary counterparts, particularly on the long-tail
distribution. Furthermore, OpenMask3D goes beyond the limitations of
close-vocabulary approaches, and enables the segmentation of object instances
based on free-form queries describing object properties such as semantics,
geometry, affordances, and material properties.

本研究介绍了一种开放词汇的 3D 实例分割任务，使用零样本学习来提高模型在目标实例分割方面的泛化性，并设计了 OpenMask3D 方法，该方法使用预测的类别不可知的 3D 实例掩码来聚合每个掩码的特征，通过多视图融合和基于 CLIP 的图像嵌入来提高分割的性能。