Recent works on open-vocabulary 3D instance segmentation show strong promise,
but at the cost of slow inference speed and high computation requirements. This
high computation cost is typically due to their heavy reliance on 3D clip
features, which require computationally expensive 2D foundation models like
Segment Anything (SAM) and CLIP for multi-view aggregation into 3D. As a
consequence, this hampers their applicability in many real-world applications
that require both fast and accurate predictions. To this end, we propose a fast
yet accurate open-vocabulary 3D instance segmentation approach, named Open-YOLO
3D, that effectively leverages only 2D object detection from multi-view RGB
images for open-vocabulary 3D instance segmentation. We address this task by
generating class-agnostic 3D masks for objects in the scene and associating
them with text prompts. We observe that the projection of class-agnostic 3D
point cloud instances already holds instance information; thus, using SAM might
only result in redundancy that unnecessarily increases the inference time. We
empirically find that a better performance of matching text prompts to 3D masks
can be achieved in a faster fashion with a 2D object detector. We validate our
Open-YOLO 3D on two benchmarks, ScanNet200 and Replica, under two scenarios:
(i) with ground truth masks, where labels are required for given object
proposals, and (ii) with class-agnostic 3D proposals generated from a 3D
proposal network. Our Open-YOLO 3D achieves state-of-the-art performance on
both datasets while obtaining up to $\sim$16$\times$ speedup compared to the
best existing method in literature. On ScanNet200 val. set, our Open-YOLO 3D
achieves mean average precision (mAP) of 24.7\% while operating at 22 seconds
per scene. Code and model are available at github.com/aminebdj/OpenYOLO3D.

我们提出了一种名为 Open-YOLO 3D 的快速而准确的开放词汇 3D 实例分割方法，在多视角 RGB 图像中有效利用仅 2D 对象检测进行开放词汇 3D 实例分割，并通过生成类无关的 3D 掩码并与文本提示相关联来处理此任务。我们的 Open-YOLO 3D 在两个基准测试集 ScanNet200 和 Replica 上实现了最先进的性能，在与文献中最佳方法相比速度提高了多达 16 倍。