Current 3D open-vocabulary scene understanding methods mostly utilize
well-aligned 2D images as the bridge to learn 3D features with language.
However, applying these approaches becomes challenging in scenarios where 2D
images are absent. In this work, we introduce a completely new pipeline,
namely, OpenIns3D, which requires no 2D image inputs, for 3D open-vocabulary
scene understanding at the instance level. The OpenIns3D framework employs a
"Mask-Snap-Lookup" scheme. The "Mask" module learns class-agnostic mask
proposals in 3D point clouds. The "Snap" module generates synthetic scene-level
images at multiple scales and leverages 2D vision language models to extract
interesting objects. The "Lookup" module searches through the outcomes of
"Snap" with the help of Mask2Pixel maps, which contain the precise
correspondence between 3D masks and synthetic images, to assign category names
to the proposed masks. This 2D input-free, easy-to-train, and flexible approach
achieved state-of-the-art results on a wide range of indoor and outdoor
datasets with a large margin. Furthermore, OpenIns3D allows for effortless
switching of 2D detectors without re-training. When integrated with
state-of-the-art 2D open-world models such as ODISE and GroundingDINO, superb
results are observed on open-vocabulary instance segmentation. When integrated
with LLM-powered 2D models like LISA, it demonstrates a remarkable capacity to
process highly complex text queries, including those that require intricate
reasoning and world knowledge. The code and model will be made publicly
available.

OpenIns3D 是一个不需要 2D 图像输入的用于实例级别的三维开放词汇场景理解的全新框架，通过 Mask-Snap-Lookup 方案实现，能够在室内和室外的广泛数据集上取得优越的结果，并支持无需重新训练即可切换 2D 检测器的功能。