3D scene representations have gained immense popularity in recent years.
Methods that use Neural Radiance fields are versatile for traditional tasks
such as novel view synthesis. In recent times, some work has emerged that aims
to extend the functionality of NeRF beyond view synthesis, for semantically
aware tasks such as editing and segmentation using 3D feature field
distillation from 2D foundation models. However, these methods have two major
limitations: (a) they are limited by the rendering speed of NeRF pipelines, and
(b) implicitly represented feature fields suffer from continuity artifacts
reducing feature quality. Recently, 3D Gaussian Splatting has shown
state-of-the-art performance on real-time radiance field rendering. In this
work, we go one step further: in addition to radiance field rendering, we
enable 3D Gaussian splatting on arbitrary-dimension semantic features via 2D
foundation model distillation. This translation is not straightforward: naively
incorporating feature fields in the 3DGS framework leads to warp-level
divergence. We propose architectural and training changes to efficiently avert
this problem. Our proposed method is general, and our experiments showcase
novel view semantic segmentation, language-guided editing and segment anything
through learning feature fields from state-of-the-art 2D foundation models such
as SAM and CLIP-LSeg. Across experiments, our distillation method is able to
provide comparable or better results, while being significantly faster to both
train and render. Additionally, to the best of our knowledge, we are the first
method to enable point and bounding-box prompting for radiance field
manipulation, by leveraging the SAM model. Project website at:
this https URL

通过 2D 基础模型的 3D 特征场蒸馏，我们提出了一种通用的方法，用于实现新颖视图的语义分割、语言引导编辑和任意维度语义特征的 3D 高斯喷洒，提供可比较或更好的结果，并且训练和渲染速度显著更快。