The integration of visual-tactile stimulus is common while humans performing
daily tasks. In contrast, using unimodal visual or tactile perception limits
the perceivable dimensionality of a subject. However, it remains a challenge to
integrate the visual and tactile perception to facilitate robotic tasks. In
this paper, we propose a novel framework for the cross-modal sensory data
generation for visual and tactile perception. Taking texture perception as an
example, we apply conditional generative adversarial networks to generate
pseudo visual images or tactile outputs from data of the other modality.
Extensive experiments on the ViTac dataset of cloth textures show that the
proposed method can produce realistic outputs from other sensory inputs. We
adopt the structural similarity index to evaluate similarity of the generated
output and real data and results show that realistic data have been generated.
Classification evaluation has also been performed to show that the inclusion of
generated data can improve the perception performance. The proposed framework
has potential to expand datasets for classification tasks, generate sensory
outputs that are not easy to access, and also advance integrated visual-tactile
perception.

本文提出了一种新的框架，利用条件生成对抗网络生成视觉或触觉图像，以实现视觉和触觉知觉的跨模态感知，并在 ViTac 数据集上进行了广泛实验，结果表明该方法能够生成逼真的数据并且有潜力扩大分类任务的数据集，生成不易获取的感官输出，并推进视觉 - 触觉知觉的整合。