Humans perceive the world using multi-modal sensory inputs such as vision,
audition, and touch. In this work, we investigate the cross-modal connection
between vision and touch. The main challenge in this cross-domain modeling task
lies in the significant scale discrepancy between the two: while our eyes
perceive an entire visual scene at once, humans can only feel a small region of
an object at any given moment. To connect vision and touch, we introduce new
tasks of synthesizing plausible tactile signals from visual inputs as well as
imagining how we interact with objects given tactile data as input. To
accomplish our goals, we first equip robots with both visual and tactile
sensors and collect a large-scale dataset of corresponding vision and tactile
image sequences. To close the scale gap, we present a new conditional
adversarial model that incorporates the scale and location information of the
touch. Human perceptual studies demonstrate that our model can produce
realistic visual images from tactile data and vice versa. Finally, we present
both qualitative and quantitative experimental results regarding different
system designs, as well as visualizing the learned representations of our
model.

本文通过在机器人装备视觉和触觉传感器，收集大规模视觉和触觉图像序列数据，使用条件对抗式模型，实现视觉和触觉的交叉连接，来合成视觉数据和触觉信号，并想象人与物体的互动。