Predicting the future interaction of objects when they come into contact with
their environment is key for autonomous agents to take intelligent and
anticipatory actions. This paper presents a perception framework that fuses
visual and tactile feedback to make predictions about the expected motion of
objects in dynamic scenes. Visual information captures object properties such
as 3D shape and location, while tactile information provides critical cues
about interaction forces and resulting object motion when it makes contact with
the environment. Utilizing a novel See-Through-your-Skin (STS) sensor that
provides high resolution multimodal sensing of contact surfaces, our system
captures both the visual appearance and the tactile properties of objects. We
interpret the dual stream signals from the sensor using a Multimodal
Variational Autoencoder (MVAE), allowing us to capture both modalities of
contacting objects and to develop a mapping from visual to tactile interaction
and vice-versa. Additionally, the perceptual system can be used to infer the
outcome of future physical interactions, which we validate through simulated
and real-world experiments in which the resting state of an object is predicted
from given initial conditions.

本文提出了一种感知框架，通过融合视觉和触觉反馈来预测动态场景中物体的运动，该框架利用一种新型的 STS 传感器来捕捉物体的视觉外观和触觉特性，利用多模态 VAE 将两种模态结合起来，可以推断未来物理交互的结果。