Executing contact-rich manipulation tasks necessitates the fusion of tactile
and visual feedback. However, the distinct nature of these modalities poses
significant challenges. In this paper, we introduce a system that leverages
visual and tactile sensory inputs to enable dexterous in-hand manipulation.
Specifically, we propose Robot Synesthesia, a novel point cloud-based tactile
representation inspired by human tactile-visual synesthesia. This approach
allows for the simultaneous and seamless integration of both sensory inputs,
offering richer spatial information and facilitating better reasoning about
robot actions. The method, trained in a simulated environment and then deployed
to a real robot, is applicable to various in-hand object rotation tasks.
Comprehensive ablations are performed on how the integration of vision and
touch can improve reinforcement learning and Sim2Real performance. Our project
page is available at this https URL .

使用视觉和触觉感知输入实现灵巧的手中操作是一项具有挑战性的任务，本文提出了一种基于点云的触觉表示方法 Robot Synesthesia，该方法通过同时无缝地整合视觉和触觉输入，提供更丰富的空间信息，有助于更好地推理机器人动作，通过在模拟环境中训练并应用于真实机器人，可适用于各种手中物体旋转任务，并通过综合消融实验验证了视觉和触觉的整合如何改善强化学习和实验到真实场景的性能。