This work presents an instance-agnostic learning framework that fuses vision
with dynamics to simultaneously learn shape, pose trajectories and physical
properties via the use of geometry as a shared representation. Unlike many
contact learning approaches that assume motion capture input and a known shape
prior for the collision model, our proposed framework learns an object's
geometric and dynamic properties from RGBD video, without requiring either
category-level or instance-level shape priors. We integrate a vision system,
BundleSDF, with a dynamics system, ContactNets and propose a cyclic training
pipeline to use the output from the dynamics module to refine the poses and the
geometry from the vision module, using perspective reprojection. Experiments
demonstrate our framework's ability to learn the geometry and dynamics of rigid
and convex objects and improve upon the current tracking framework.

通过使用几何作为共享表示，本研究提出了一种实例不可知的学习框架，融合了视觉与动力学，以同时学习形状、姿态轨迹和物理属性。通过使用 RGBD 视频，无需先验知识和已知形状，该框架能够学习对象的几何和动力学特性。实验证明了该框架在学习刚性和凸物体的几何和动力学方面的能力，并且改进了当前的跟踪框架。