This paper introduces OpenGaussian, a method based on 3D Gaussian Splatting
(3DGS) capable of 3D point-level open vocabulary understanding. Our primary
motivation stems from observing that existing 3DGS-based open vocabulary
methods mainly focus on 2D pixel-level parsing. These methods struggle with 3D
point-level tasks due to weak feature expressiveness and inaccurate 2D-3D
feature associations. To ensure robust feature presentation and 3D point-level
understanding, we first employ SAM masks without cross-frame associations to
train instance features with 3D consistency. These features exhibit both
intra-object consistency and inter-object distinction. Then, we propose a
two-stage codebook to discretize these features from coarse to fine levels. At
the coarse level, we consider the positional information of 3D points to
achieve location-based clustering, which is then refined at the fine level.
Finally, we introduce an instance-level 3D-2D feature association method that
links 3D points to 2D masks, which are further associated with 2D CLIP
features. Extensive experiments, including open vocabulary-based 3D object
selection, 3D point cloud understanding, click-based 3D object selection, and
ablation studies, demonstrate the effectiveness of our proposed method. Project
page: this https URL

这篇论文介绍了 OpenGaussian，一种基于 3D 高斯散射（3DGS）的方法，能够实现 3D 点级别的开放性词汇理解。通过使用 SAM 掩码和具有 3D 一致性的实例特征训练以及引入实例级别的 3D-2D 特征关联方法，该方法成功地实现了 3D 对象选择和理解。