Open-vocabulary 3D scene understanding presents a significant challenge in
computer vision, withwide-ranging applications in embodied agents and augmented
reality systems. Previous approaches haveadopted Neural Radiance Fields (NeRFs)
to analyze 3D scenes. In this paper, we introduce SemanticGaussians, a novel
open-vocabulary scene understanding approach based on 3D Gaussian Splatting.
Our keyidea is distilling pre-trained 2D semantics into 3D Gaussians. We design
a versatile projection approachthat maps various 2Dsemantic features from
pre-trained image encoders into a novel semantic component of 3D Gaussians,
withoutthe additional training required by NeRFs. We further build a 3D
semantic network that directly predictsthe semantic component from raw 3D
Gaussians for fast inference. We explore several applications ofSemantic
Gaussians: semantic segmentation on ScanNet-20, where our approach attains a
4.2% mIoU and 4.0%mAcc improvement over prior open-vocabulary scene
understanding counterparts; object part segmentation,sceneediting, and
spatial-temporal segmentation with better qualitative results over 2D and 3D
baselines,highlighting its versatility and effectiveness on supporting diverse
downstream tasks.

基于 3D 高斯喷斑的新型开放词汇场景理解方法，通过提取预训练的 2D 语义学习特征，将其融入 3D 高斯成分，并构建 3D 高斯语义网络用于快速推断，实现了在 ScanNet-20 上对语义分割的改进、对物体部分分割、场景编辑以及时空分割的多样性支持和优越性。