While neural implicit representations have gained popularity in multi-view 3D
reconstruction, previous work struggles to yield physically plausible results,
thereby limiting their applications in physics-demanding domains like embodied
AI and robotics. The lack of plausibility originates from both the absence of
physics modeling in the existing pipeline and their inability to recover
intricate geometrical structures. In this paper, we introduce PhyRecon, which
stands as the first approach to harness both differentiable rendering and
differentiable physics simulation to learn implicit surface representations.
Our framework proposes a novel differentiable particle-based physical simulator
seamlessly integrated with the neural implicit representation. At its core is
an efficient transformation between SDF-based implicit representation and
explicit surface points by our proposed algorithm, Surface Points Marching
Cubes (SP-MC), enabling differentiable learning with both rendering and
physical losses. Moreover, we model both rendering and physical uncertainty to
identify and compensate for the inconsistent and inaccurate monocular geometric
priors. The physical uncertainty additionally enables a physics-guided pixel
sampling to enhance the learning of slender structures. By amalgamating these
techniques, our model facilitates efficient joint modeling with appearance,
geometry, and physics. Extensive experiments demonstrate that PhyRecon
significantly outperforms all state-of-the-art methods in terms of
reconstruction quality. Our reconstruction results also yield superior physical
stability, verified by Isaac Gym, with at least a 40% improvement across all
datasets, opening broader avenues for future physics-based applications.

该论文介绍了使用可微渲染和可微物理模拟相结合的方法来学习隐式表面表示，并通过改进的算法实现了隐式表示和显式表面点之间的有效转化，从而在物理要求高的领域中获得更加真实和准确的重建结果。