Open-vocabulary querying in 3D space is challenging but essential for scene
understanding tasks such as object localization and segmentation.
Language-embedded scene representations have made progress by incorporating
language features into 3D spaces. However, their efficacy heavily depends on
neural networks that are resource-intensive in training and rendering. Although
recent 3D Gaussians offer efficient and high-quality novel view synthesis,
directly embedding language features in them leads to prohibitive memory usage
and decreased performance. In this work, we introduce Language Embedded 3D
Gaussians, a novel scene representation for open-vocabulary query tasks.
Instead of embedding high-dimensional raw semantic features on 3D Gaussians, we
propose a dedicated quantization scheme that drastically alleviates the memory
requirement, and a novel embedding procedure that achieves smoother yet high
accuracy query, countering the multi-view feature inconsistencies and the
high-frequency inductive bias in point-based representations. Our comprehensive
experiments show that our representation achieves the best visual quality and
language querying accuracy across current language-embedded representations,
while maintaining real-time rendering frame rates on a single desktop GPU.

通过使用一种新颖的场景表示方法 —— 嵌入语言的 3D 高斯模型，本研究在 3D 空间中实现了开放词汇查询任务，取得了最佳的视觉质量和语言查询准确性，并且保持在单个台式 GPU 上的实时渲染帧速率。

嵌入式语言的三维高斯函数用于开放词汇场景理解

Language Embedded 3D Gaussians for Open-Vocabulary Scene Understanding

In this paper, we address the challenge of decomposing Neural Radiance Fields
(NeRF) into objects from an open vocabulary, a critical task for object
manipulation in 3D reconstruction and view synthesis. Current techniques for
NeRF decomposition involve a trade-off between the flexibility of processing
open-vocabulary queries and the accuracy of 3D segmentation. We present,
Open-vocabulary Embedded Neural Radiance Fields (Open-NeRF), that leverage
large-scale, off-the-shelf, segmentation models like the Segment Anything Model
(SAM) and introduce an integrate-and-distill paradigm with hierarchical
embeddings to achieve both the flexibility of open-vocabulary querying and 3D
segmentation accuracy. Open-NeRF first utilizes large-scale foundation models
to generate hierarchical 2D mask proposals from varying viewpoints. These
proposals are then aligned via tracking approaches and integrated within the 3D
space and subsequently distilled into the 3D field. This process ensures
consistent recognition and granularity of objects from different viewpoints,
even in challenging scenarios involving occlusion and indistinct features. Our
experimental results show that the proposed Open-NeRF outperforms
state-of-the-art methods such as LERF \cite{lerf} and FFD \cite{ffd} in
open-vocabulary scenarios. Open-NeRF offers a promising solution to NeRF
decomposition, guided by open-vocabulary queries, enabling novel applications
in robotics and vision-language interaction in open-world 3D scenes.

该研究关注解决从开放词汇中将神经辐射场（NeRF）分解为对象的挑战，这对于三维重建和视图合成中的对象操作是至关重要的。我们提出了 Open-NeRF，利用大规模、现成的分割模型，如 Segment Anything Model（SAM），并引入了一种集成和蒸馏范式，通过层次嵌入来既实现开放词汇查询的灵活性，又保持三维分割的准确性。Open-NeRF 首先利用大规模的基础模型根据不同视角生成分层的二维掩模提案，然后通过跟踪方法对这些提案进行对齐，并在三维空间中进行集成，最后蒸馏为三维场。该过程确保了不同视角下的对象一致识别和细粒度，即使在涉及遮挡和模糊特征的挑战性场景中也是如此。实验结果表明，Open-NeRF 在开放词汇的场景中胜过了 LERF 和 FFD 等最先进的方法。Open-NeRF 为 NeRF 分解提供了一个有前途的解决方案，通过开放词汇查询引导，能够在开放世界的三维场景中实现新的机器人和视觉语言交互应用。