3D open-vocabulary scene understanding, crucial for advancing augmented
reality and robotic applications, involves interpreting and locating specific
regions within a 3D space as directed by natural language instructions. To this
end, we introduce GOI, a framework that integrates semantic features from 2D
vision-language foundation models into 3D Gaussian Splatting (3DGS) and
identifies 3D Gaussians of Interest using an Optimizable Semantic-space
Hyperplane. Our approach includes an efficient compression method that utilizes
scene priors to condense noisy high-dimensional semantic features into compact
low-dimensional vectors, which are subsequently embedded in 3DGS. During the
open-vocabulary querying process, we adopt a distinct approach compared to
existing methods, which depend on a manually set fixed empirical threshold to
select regions based on their semantic feature distance to the query text
embedding. This traditional approach often lacks universal accuracy, leading to
challenges in precisely identifying specific target areas. Instead, our method
treats the feature selection process as a hyperplane division within the
feature space, retaining only those features that are highly relevant to the
query. We leverage off-the-shelf 2D Referring Expression Segmentation (RES)
models to fine-tune the semantic-space hyperplane, enabling a more precise
distinction between target regions and others. This fine-tuning substantially
improves the accuracy of open-vocabulary queries, ensuring the precise
localization of pertinent 3D Gaussians. Extensive experiments demonstrate GOI's
superiority over previous state-of-the-art methods. Our project page is
available at this https URL .

通过在 3D 空间中解释和定位特定区域，GOI 整合了来自 2D 视觉 - 语言基础模型的语义特征，利用可优化的语义空间超平面识别感兴趣的 3D 高斯。通过在特性空间内进行超平面划分的特征选择过程，GOI 方法通过微调语义空间超平面，实现了开放式词汇查询的精准定位。大量实验证明了 GOI 方法相对于之前的最先进方法的优越性。