The recent Gaussian Splatting achieves high-quality and real-time novel-view
synthesis of the 3D scenes. However, it is solely concentrated on the
appearance and geometry modeling, while lacking in fine-grained object-level
scene understanding. To address this issue, we propose Gaussian Grouping, which
extends Gaussian Splatting to jointly reconstruct and segment anything in
open-world 3D scenes. We augment each Gaussian with a compact Identity
Encoding, allowing the Gaussians to be grouped according to their object
instance or stuff membership in the 3D scene. Instead of resorting to expensive
3D labels, we supervise the Identity Encodings during the differentiable
rendering by leveraging the 2D mask predictions by SAM, along with introduced
3D spatial consistency regularization. Comparing to the implicit NeRF
representation, we show that the discrete and grouped 3D Gaussians can
reconstruct, segment and edit anything in 3D with high visual quality, fine
granularity and efficiency. Based on Gaussian Grouping, we further propose a
local Gaussian Editing scheme, which shows efficacy in versatile scene editing
applications, including 3D object removal, inpainting, colorization and scene
recomposition. Our code and models will be at
this https URL

通过识别每个高斯函数的紧凑身份编码，我们将高斯函数根据其对象实例或物体成员资格在三维场景中分组，从而实现了高质量的三维重建、分割和编辑，同时通过 SAM 的二维掩模预测和引入的三维空间一致性正则化来监督身份编码，通过 Gaussian Grouping 进一步提出了一种本地高斯编辑方案，用于 3D 对象移除、修复、上色和场景重组等多种场景编辑应用。

高斯分组：对 3D 场景中的物体进行分割和编辑

Gaussian Grouping: Segment and Edit Anything in 3D Scenes

Robots need object-level scene understanding to manipulate objects while
reasoning about contact, support, and occlusion among objects. Given a pile of
objects, object recognition and reconstruction can identify the boundary of
object instances, giving important cues as to how the objects form and support
the pile. In this work, we present a system, SafePicking, that integrates
object-level mapping and learning-based motion planning to generate a motion
that safely extracts occluded target objects from a pile. Planning is done by
learning a deep Q-network that receives observations of predicted poses and a
depth-based heightmap to output a motion trajectory, trained to maximize a
safety metric reward. Our results show that the observation fusion of poses and
depth-sensing gives both better performance and robustness to the model. We
evaluate our methods using the YCB objects in both simulation and the real
world, achieving safe object extraction from piles.

本文介绍了一个名为 SafePicking 的系统，它结合了目标级别的建图和学习驱动的运动规划来安全地从堆中提取目标物体，通过使用模拟和真实世界中的 YCB 物体进行评估，结果表明安全度量奖励的深度 Q 网络的学习可以实现目标提取的成功和稳健性提高。