3D open-vocabulary scene understanding, crucial for advancing augmented
reality and robotic applications, involves interpreting and locating specific
regions within a 3D space as directed by natural language instructions. To this
end, we introduce GOI, a framework that integrates semantic features from 2D
vision-language foundation models into 3D Gaussian Splatting (3DGS) and
identifies 3D Gaussians of Interest using an Optimizable Semantic-space
Hyperplane. Our approach includes an efficient compression method that utilizes
scene priors to condense noisy high-dimensional semantic features into compact
low-dimensional vectors, which are subsequently embedded in 3DGS. During the
open-vocabulary querying process, we adopt a distinct approach compared to
existing methods, which depend on a manually set fixed empirical threshold to
select regions based on their semantic feature distance to the query text
embedding. This traditional approach often lacks universal accuracy, leading to
challenges in precisely identifying specific target areas. Instead, our method
treats the feature selection process as a hyperplane division within the
feature space, retaining only those features that are highly relevant to the
query. We leverage off-the-shelf 2D Referring Expression Segmentation (RES)
models to fine-tune the semantic-space hyperplane, enabling a more precise
distinction between target regions and others. This fine-tuning substantially
improves the accuracy of open-vocabulary queries, ensuring the precise
localization of pertinent 3D Gaussians. Extensive experiments demonstrate GOI's
superiority over previous state-of-the-art methods. Our project page is
available at this https URL .

通过在 3D 空间中解释和定位特定区域，GOI 整合了来自 2D 视觉 - 语言基础模型的语义特征，利用可优化的语义空间超平面识别感兴趣的 3D 高斯。通过在特性空间内进行超平面划分的特征选择过程，GOI 方法通过微调语义空间超平面，实现了开放式词汇查询的精准定位。大量实验证明了 GOI 方法相对于之前的最先进方法的优越性。

使用可优化的开放词汇语义空间超平面找到感兴趣的三维高斯函数

GOI: Find 3D Gaussians of Interest with an Optimizable Open-vocabulary  Semantic-space Hyperplane

3D open-vocabulary scene understanding aims to recognize arbitrary novel
categories beyond the base label space. However, existing works not only fail
to fully utilize all the available modal information in the 3D domain but also
lack sufficient granularity in representing the features of each modality. In
this paper, we propose a unified multimodal 3D open-vocabulary scene
understanding network, namely UniM-OV3D, which aligns point clouds with image,
language and depth. To better integrate global and local features of the point
clouds, we design a hierarchical point cloud feature extraction module that
learns comprehensive fine-grained feature representations. Further, to
facilitate the learning of coarse-to-fine point-semantic representations from
captions, we propose the utilization of hierarchical 3D caption pairs,
capitalizing on geometric constraints across various viewpoints of 3D scenes.
Extensive experimental results demonstrate the effectiveness and superiority of
our method in open-vocabulary semantic and instance segmentation, which
achieves state-of-the-art performance on both indoor and outdoor benchmarks
such as ScanNet, ScanNet200, S3IDS and nuScenes. Code is available at
this https URL

本文提出了一种统一的多模态三维开放词汇场景理解网络 (UniM-OV3D)，通过对齐点云、图像、语言和深度信息，设计了一个分层点云特征提取模块并使用了层次化三维标题对来提高粗到细点云语义表示学习，实验结果表明我们的方法在室内外基准测试中实现了开放式语义和实例分割的最先进性能。

UniM-OV3D：使用细粒度特征表示的单模态开放词汇三维场景理解

UniM-OV3D: Uni-Modality Open-Vocabulary 3D Scene Understanding with  Fine-Grained Feature Representation

Current 3D open-vocabulary scene understanding methods mostly utilize
well-aligned 2D images as the bridge to learn 3D features with language.
However, applying these approaches becomes challenging in scenarios where 2D
images are absent. In this work, we introduce a completely new pipeline,
namely, OpenIns3D, which requires no 2D image inputs, for 3D open-vocabulary
scene understanding at the instance level. The OpenIns3D framework employs a
"Mask-Snap-Lookup" scheme. The "Mask" module learns class-agnostic mask
proposals in 3D point clouds. The "Snap" module generates synthetic scene-level
images at multiple scales and leverages 2D vision language models to extract
interesting objects. The "Lookup" module searches through the outcomes of
"Snap" with the help of Mask2Pixel maps, which contain the precise
correspondence between 3D masks and synthetic images, to assign category names
to the proposed masks. This 2D input-free, easy-to-train, and flexible approach
achieved state-of-the-art results on a wide range of indoor and outdoor
datasets with a large margin. Furthermore, OpenIns3D allows for effortless
switching of 2D detectors without re-training. When integrated with
state-of-the-art 2D open-world models such as ODISE and GroundingDINO, superb
results are observed on open-vocabulary instance segmentation. When integrated
with LLM-powered 2D models like LISA, it demonstrates a remarkable capacity to
process highly complex text queries, including those that require intricate
reasoning and world knowledge. The code and model will be made publicly
available.

OpenIns3D 是一个不需要 2D 图像输入的用于实例级别的三维开放词汇场景理解的全新框架，通过 Mask-Snap-Lookup 方案实现，能够在室内和室外的广泛数据集上取得优越的结果，并支持无需重新训练即可切换 2D 检测器的功能。