Recent works on open-vocabulary 3D instance segmentation show strong promise,
but at the cost of slow inference speed and high computation requirements. This
high computation cost is typically due to their heavy reliance on 3D clip
features, which require computationally expensive 2D foundation models like
Segment Anything (SAM) and CLIP for multi-view aggregation into 3D. As a
consequence, this hampers their applicability in many real-world applications
that require both fast and accurate predictions. To this end, we propose a fast
yet accurate open-vocabulary 3D instance segmentation approach, named Open-YOLO
3D, that effectively leverages only 2D object detection from multi-view RGB
images for open-vocabulary 3D instance segmentation. We address this task by
generating class-agnostic 3D masks for objects in the scene and associating
them with text prompts. We observe that the projection of class-agnostic 3D
point cloud instances already holds instance information; thus, using SAM might
only result in redundancy that unnecessarily increases the inference time. We
empirically find that a better performance of matching text prompts to 3D masks
can be achieved in a faster fashion with a 2D object detector. We validate our
Open-YOLO 3D on two benchmarks, ScanNet200 and Replica, under two scenarios:
(i) with ground truth masks, where labels are required for given object
proposals, and (ii) with class-agnostic 3D proposals generated from a 3D
proposal network. Our Open-YOLO 3D achieves state-of-the-art performance on
both datasets while obtaining up to $\sim$16$\times$ speedup compared to the
best existing method in literature. On ScanNet200 val. set, our Open-YOLO 3D
achieves mean average precision (mAP) of 24.7\% while operating at 22 seconds
per scene. Code and model are available at github.com/aminebdj/OpenYOLO3D.

我们提出了一种名为 Open-YOLO 3D 的快速而准确的开放词汇 3D 实例分割方法，在多视角 RGB 图像中有效利用仅 2D 对象检测进行开放词汇 3D 实例分割，并通过生成类无关的 3D 掩码并与文本提示相关联来处理此任务。我们的 Open-YOLO 3D 在两个基准测试集 ScanNet200 和 Replica 上实现了最先进的性能，在与文献中最佳方法相比速度提高了多达 16 倍。

Open-YOLO 3D：面向快速准确的开放词汇的 3D 实例分割

Open-YOLO 3D: Towards Fast and Accurate Open-Vocabulary 3D Instance  Segmentation

In this paper, we investigate Open-Vocabulary 3D Instance Segmentation
(OV-3DIS) with free-form language instructions. Earlier works that rely on only
annotated base categories for training suffer from limited generalization to
unseen novel categories. Recent works mitigate poor generalizability to novel
categories by generating class-agnostic masks or projecting generalized masks
from 2D to 3D, but disregard semantic or geometry information, leading to
sub-optimal performance. Instead, generating generalizable but semantic-related
masks directly from 3D point clouds would result in superior outcomes. In this
paper, we introduce Segment any 3D Object with LanguagE (SOLE), which is a
semantic and geometric-aware visual-language learning framework with strong
generalizability by generating semantic-related masks directly from 3D point
clouds. Specifically, we propose a multimodal fusion network to incorporate
multimodal semantics in both backbone and decoder. In addition, to align the 3D
segmentation model with various language instructions and enhance the mask
quality, we introduce three types of multimodal associations as supervision.
Our SOLE outperforms previous methods by a large margin on ScanNetv2,
ScanNet200, and Replica benchmarks, and the results are even close to the
fully-supervised counterpart despite the absence of class annotations in the
training. Furthermore, extensive qualitative results demonstrate the
versatility of our SOLE to language instructions.

本文介绍了一种基于语言指令的语义和几何感知的视觉语言学习框架，通过直接从 3D 点云生成语义相关的掩码，从而在 ScanNetv2、ScanNet200 和 Replica 基准测试中明显优于现有方法，且结果接近全监督对照组，尽管训练数据中缺乏类别标注。

使用语言对任意 3D 物体进行分割

Segment Any 3D Object with Language

We introduce the task of open-vocabulary 3D instance segmentation.
Traditional approaches for 3D instance segmentation largely rely on existing 3D
annotated datasets, which are restricted to a closed-set of object categories.
This is an important limitation for real-life applications where one might need
to perform tasks guided by novel, open-vocabulary queries related to objects
from a wide variety. Recently, open-vocabulary 3D scene understanding methods
have emerged to address this problem by learning queryable features per each
point in the scene. While such a representation can be directly employed to
perform semantic segmentation, existing methods have limitations in their
ability to identify object instances. In this work, we address this limitation,
and propose OpenMask3D, which is a zero-shot approach for open-vocabulary 3D
instance segmentation. Guided by predicted class-agnostic 3D instance masks,
our model aggregates per-mask features via multi-view fusion of CLIP-based
image embeddings. We conduct experiments and ablation studies on the ScanNet200
dataset to evaluate the performance of OpenMask3D, and provide insights about
the open-vocabulary 3D instance segmentation task. We show that our approach
outperforms other open-vocabulary counterparts, particularly on the long-tail
distribution. Furthermore, OpenMask3D goes beyond the limitations of
close-vocabulary approaches, and enables the segmentation of object instances
based on free-form queries describing object properties such as semantics,
geometry, affordances, and material properties.

本研究介绍了一种开放词汇的 3D 实例分割任务，使用零样本学习来提高模型在目标实例分割方面的泛化性，并设计了 OpenMask3D 方法，该方法使用预测的类别不可知的 3D 实例掩码来聚合每个掩码的特征，通过多视图融合和基于 CLIP 的图像嵌入来提高分割的性能。