Physics-based understanding of object interactions from sensory observations is an essential capability in augmented reality and robotics. It enables capturing the properties of a scene for simulation and control. In this paper, we propose a novel approach for real-to-sim which tracks rigid objects in 3D from RGB-D images and infers physical properties of the objects. We use a differentiable physics simulation as state-transition model in an Extended Kalman Filter which can model contact and friction for arbitrary mesh-based shapes and in this way estimate physically plausible trajectories. We demonstrate that our approach can filter position, orientation, velocities, and concurrently can estimate the coefficient of friction of the objects. We analyse our approach on various sliding scenarios in synthetic image sequences of single objects and colliding objects. We also demonstrate and evaluate our approach on a real-world dataset. We will make our novel benchmark datasets publicly available to foster future research in this novel problem setting and comparison with our method.

提出了一种从RGB-D图像中跟踪三维刚体物体并推断物体物理属性的新颖方法，借助可微分物理仿真作为状态转移模型，利用扩展卡尔曼滤波器模型接触和摩擦来估计真实的运动轨迹和摩擦系数，在合成图像序列和真实世界数据集上得到了验证和评估。

基于物理模型的刚体目标跟踪与RGB-D视频摩擦滤波