Scene representations using 3D Gaussian primitives have produced excellent
results in modeling the appearance of static and dynamic 3D scenes. Many
graphics applications, however, demand the ability to manipulate both the
appearance and the physical properties of objects. We introduce Feature
Splatting, an approach that unifies physics-based dynamic scene synthesis with
rich semantics from vision language foundation models that are grounded by
natural language. Our first contribution is a way to distill high-quality,
object-centric vision-language features into 3D Gaussians, that enables
semi-automatic scene decomposition using text queries. Our second contribution
is a way to synthesize physics-based dynamics from an otherwise static scene
using a particle-based simulator, in which material properties are assigned
automatically via text queries. We ablate key techniques used in this pipeline,
to illustrate the challenge and opportunities in using feature-carrying 3D
Gaussians as a unified format for appearance, geometry, material properties and
semantics grounded on natural language. Project website:
this https URL

使用 3D 高斯原始物体来建模静态和动态 3D 场景的外观已经取得了优秀的结果。本研究提出了一种统一物理动态场景合成和由自然语言支持的丰富语义的特征平面化方法。通过将高质量的面向对象视觉语言特征整合为 3D 高斯模型，能够利用文本查询实现半自动场景分解。同时，通过基于粒子的模拟器从静态场景中合成物理动力学，通过文本查询自动分配材料属性。本研究在这一流程中消去了关键技术，以说明将携带特征的 3D 高斯模型作为外观、几何、材料属性和基于自然语言的语义的统一格式的挑战和机遇。

特征投射与基于语言的基于物理的场景合成和编辑

Feature Splatting: Language-Driven Physics-Based Scene Synthesis and  Editing

Real-life control tasks involve matters of various substances---rigid or soft
bodies, liquid, gas---each with distinct physical behaviors. This poses
challenges to traditional rigid-body physics engines. Particle-based simulators
have been developed to model the dynamics of these complex scenes; however,
relying on approximation techniques, their simulation often deviates from
real-world physics, especially in the long term. In this paper, we propose to
learn a particle-based simulator for complex control tasks. Combining learning
with particle-based systems brings in two major benefits: first, the learned
simulator, just like other particle-based systems, acts widely on objects of
different materials; second, the particle-based representation poses strong
inductive bias for learning: particles of the same type have the same dynamics
within. This enables the model to quickly adapt to new environments of unknown
dynamics within a few observations. We demonstrate robots achieving complex
manipulation tasks using the learned simulator, such as manipulating fluids and
deformable foam, with experiments both in simulation and in the real world. Our
study helps lay the foundation for robot learning of dynamic scenes with
particle-based representations.

本文提出了一种基于粒子的学习模拟器来解决复杂控制问题，同时在模拟跨物质的复杂场景方面具有很好的表现，使得机器人能够完成复杂的操控任务，如操纵流体和可形变泡沫，利用在仿真及现实中的实验结果显示了该方法的可行性。