Tactile feedback is critical for understanding the dynamics of both rigid and
deformable objects in many manipulation tasks, such as non-prehensile
manipulation and dense packing. We introduce an approach that combines visual
and tactile sensing for robotic manipulation by learning a neural,
tactile-informed dynamics model. Our proposed framework, RoboPack, employs a
recurrent graph neural network to estimate object states, including particles
and object-level latent physics information, from historical visuo-tactile
observations and to perform future state predictions. Our tactile-informed
dynamics model, learned from real-world data, can solve downstream robotics
tasks with model-predictive control. We demonstrate our approach on a real
robot equipped with a compliant Soft-Bubble tactile sensor on non-prehensile
manipulation and dense packing tasks, where the robot must infer the physics
properties of objects from direct and indirect interactions. Trained on only an
average of 30 minutes of real-world interaction data per task, our model can
perform online adaptation and make touch-informed predictions. Through
extensive evaluations in both long-horizon dynamics prediction and real-world
manipulation, our method demonstrates superior effectiveness compared to
previous learning-based and physics-based simulation systems.

利用视触知觉感知和深度学习的方法来进行机器人操作，其中结合了视触反馈，通过学习神经网络模型进行物体状态估计和未来状态预测。通过使用真实世界的数据进行训练，该模型可以在在线适应中进行触觉信息驱动的预测，具有较强的有效性。