The recent Gaussian Splatting achieves high-quality and real-time novel-view
synthesis of the 3D scenes. However, it is solely concentrated on the
appearance and geometry modeling, while lacking in fine-grained object-level
scene understanding. To address this issue, we propose Gaussian Grouping, which
extends Gaussian Splatting to jointly reconstruct and segment anything in
open-world 3D scenes. We augment each Gaussian with a compact Identity
Encoding, allowing the Gaussians to be grouped according to their object
instance or stuff membership in the 3D scene. Instead of resorting to expensive
3D labels, we supervise the Identity Encodings during the differentiable
rendering by leveraging the 2D mask predictions by SAM, along with introduced
3D spatial consistency regularization. Comparing to the implicit NeRF
representation, we show that the discrete and grouped 3D Gaussians can
reconstruct, segment and edit anything in 3D with high visual quality, fine
granularity and efficiency. Based on Gaussian Grouping, we further propose a
local Gaussian Editing scheme, which shows efficacy in versatile scene editing
applications, including 3D object removal, inpainting, colorization and scene
recomposition. Our code and models will be at
this https URL

通过识别每个高斯函数的紧凑身份编码，我们将高斯函数根据其对象实例或物体成员资格在三维场景中分组，从而实现了高质量的三维重建、分割和编辑，同时通过 SAM 的二维掩模预测和引入的三维空间一致性正则化来监督身份编码，通过 Gaussian Grouping 进一步提出了一种本地高斯编辑方案，用于 3D 对象移除、修复、上色和场景重组等多种场景编辑应用。