Surgical scene simulation plays a crucial role in surgical education and
simulator-based robot learning. Traditional approaches for creating these
environments with surgical scene involve a labor-intensive process where
designers hand-craft tissues models with textures and geometries for soft body
simulations. This manual approach is not only time-consuming but also limited
in the scalability and realism. In contrast, data-driven simulation offers a
compelling alternative. It has the potential to automatically reconstruct 3D
surgical scenes from real-world surgical video data, followed by the
application of soft body physics. This area, however, is relatively uncharted.
In our research, we introduce 3D Gaussian as a learnable representation for
surgical scene, which is learned from stereo endoscopic video. To prevent
over-fitting and ensure the geometrical correctness of these scenes, we
incorporate depth supervision and anisotropy regularization into the Gaussian
learning process. Furthermore, we apply the Material Point Method, which is
integrated with physical properties, to the 3D Gaussians to achieve realistic
scene deformations. Our method was evaluated on our collected in-house and
public surgical videos datasets. Results show that it can reconstruct and
simulate surgical scenes from endoscopic videos efficiently-taking only a few
minutes to reconstruct the surgical scene-and produce both visually and
physically plausible deformations at a speed approaching real-time. The results
demonstrate great potential of our proposed method to enhance the efficiency
and variety of simulations available for surgical education and robot learning.

我们的研究介绍了一种可学习的三维高斯表示方法，通过从立体内窥镜视频中学习，可以重建和模拟手术场景，以提高手术教育和机器人学习的效率和多样性。

利用机器人手术视频的高效数据驱动场景模拟：基于物理嵌入的 3D 高斯方法

Efficient Data-driven Scene Simulation using Robotic Surgery Videos via  Physics-embedded 3D Gaussians

Differentiable physics simulation provides an avenue for tackling previously
intractable challenges through gradient-based optimization, thereby greatly
improving the efficiency of solving robotics-related problems. To apply
differentiable simulation in diverse robotic manipulation scenarios, a key
challenge is to integrate various materials in a unified framework. We present
SoftMAC, a differentiable simulation framework coupling soft bodies with
articulated rigid bodies and clothes. SoftMAC simulates soft bodies with the
continuum-mechanics-based Material Point Method (MPM). We provide a
forecast-based contact model for MPM, which greatly reduces artifacts like
penetration and unnatural rebound. To couple MPM particles with deformable and
non-volumetric clothes meshes, we also propose a penetration tracing algorithm
that reconstructs the signed distance field in local area. Based on simulators
for each modality and the contact model, we develop a differentiable coupling
mechanism to simulate the interactions between soft bodies and the other two
types of materials. Comprehensive experiments are conducted to validate the
effectiveness and accuracy of the proposed differentiable pipeline in
downstream robotic manipulation applications. Supplementary materials and
videos are available on our project website at
this https URL

通过可微分物理模拟和基于梯度的优化，SoftMAC 框架结合了软体、关节刚体和服装，有效解决了在机器人操纵中集成多种材料的关键难题，并通过实验证明了其在机器人操纵应用中的有效性和准确性。

SoftMAC：基于预测接触模型的可微软体仿真与关节刚体和服装的双向耦合

SoftMAC: Differentiable Soft Body Simulation with Forecast-based Contact  Model and Two-way Coupling with Articulated Rigid Bodies and Clothes

We introduce PhysGaussian, a new method that seamlessly integrates physically
grounded Newtonian dynamics within 3D Gaussians to achieve high-quality novel
motion synthesis. Employing a custom Material Point Method (MPM), our approach
enriches 3D Gaussian kernels with physically meaningful kinematic deformation
and mechanical stress attributes, all evolved in line with continuum mechanics
principles. A defining characteristic of our method is the seamless integration
between physical simulation and visual rendering: both components utilize the
same 3D Gaussian kernels as their discrete representations. This negates the
necessity for triangle/tetrahedron meshing, marching cubes, "cage meshes," or
any other geometry embedding, highlighting the principle of "what you see is
what you simulate (WS$^2$)." Our method demonstrates exceptional versatility
across a wide variety of materials--including elastic entities, metals,
non-Newtonian fluids, and granular materials--showcasing its strong
capabilities in creating diverse visual content with novel viewpoints and
movements. Our project page is at: this https URL

我们介绍了 PhysGaussian，一种无缝集成物理基础牛顿动力学和三维高斯函数的新方法，以实现高质量的新颖动作合成。