Surgical scene simulation plays a crucial role in surgical education and
simulator-based robot learning. Traditional approaches for creating these
environments with surgical scene involve a labor-intensive process where
designers hand-craft tissues models with textures and geometries for soft body
simulations. This manual approach is not only time-consuming but also limited
in the scalability and realism. In contrast, data-driven simulation offers a
compelling alternative. It has the potential to automatically reconstruct 3D
surgical scenes from real-world surgical video data, followed by the
application of soft body physics. This area, however, is relatively uncharted.
In our research, we introduce 3D Gaussian as a learnable representation for
surgical scene, which is learned from stereo endoscopic video. To prevent
over-fitting and ensure the geometrical correctness of these scenes, we
incorporate depth supervision and anisotropy regularization into the Gaussian
learning process. Furthermore, we apply the Material Point Method, which is
integrated with physical properties, to the 3D Gaussians to achieve realistic
scene deformations. Our method was evaluated on our collected in-house and
public surgical videos datasets. Results show that it can reconstruct and
simulate surgical scenes from endoscopic videos efficiently-taking only a few
minutes to reconstruct the surgical scene-and produce both visually and
physically plausible deformations at a speed approaching real-time. The results
demonstrate great potential of our proposed method to enhance the efficiency
and variety of simulations available for surgical education and robot learning.

我们的研究介绍了一种可学习的三维高斯表示方法，通过从立体内窥镜视频中学习，可以重建和模拟手术场景，以提高手术教育和机器人学习的效率和多样性。

利用机器人手术视频的高效数据驱动场景模拟：基于物理嵌入的 3D 高斯方法

Efficient Data-driven Scene Simulation using Robotic Surgery Videos via  Physics-embedded 3D Gaussians

Data-driven simulation is an important step-forward in computational physics
when traditional numerical methods meet their limits. Learning-based simulators
have been widely studied in past years; however, most previous works view
simulation as a general spatial-temporal prediction problem and take little
physical guidance in designing their neural network architectures. In this
paper, we introduce the alternating convolutional Long Short-Term Memory
(Alt-ConvLSTM) that models the force propagation mechanisms in a deformable
object with near-uniform material properties. Specifically, we propose an
accumulation state, and let the network update its cell state and the
accumulation state alternately. We demonstrate how this novel scheme imitates
the alternate updates of the first and second-order terms in the forward Euler
method of numerical PDE solvers. Benefiting from this, our network only
requires a small number of parameters, independent of the number of the
simulated particles, and also retains the essential features in ConvLSTM,
making it naturally applicable to sequential data with spatial inputs and
outputs. We validate our Alt-ConvLSTM on human soft tissue simulation with
thousands of particles and consistent body pose changes. Experimental results
show that Alt-ConvLSTM efficiently models the material kinetic features and
greatly outperforms vanilla ConvLSTM with only the single state update.

本文提出了一种新的交替卷积长短时记忆神经网络 (Alt-ConvLSTM) 用于模拟人类软组织动力学特性，并在实验中展现了其显著的性能优势。