Humans rely on the synergy of their senses for most essential tasks. For
tasks requiring object manipulation, we seamlessly and effectively exploit the
complementarity of our senses of vision and touch. This paper draws inspiration
from such capabilities and aims to find a systematic approach to fuse visual
and tactile information in a reinforcement learning setting. We propose Masked
Multimodal Learning (M3L), which jointly learns a policy and visual-tactile
representations based on masked autoencoding. The representations jointly
learned from vision and touch improve sample efficiency, and unlock
generalization capabilities beyond those achievable through each of the senses
separately. Remarkably, representations learned in a multimodal setting also
benefit vision-only policies at test time. We evaluate M3L on three simulated
environments with both visual and tactile observations: robotic insertion, door
opening, and dexterous in-hand manipulation, demonstrating the benefits of
learning a multimodal policy. Code and videos of the experiments are available
at this https URL

利用视觉和触觉信息在强化学习环境中联合学习策略和视觉触觉表示的蒙掩多模态学习（M3L）方法提高样本效率，并超越单个感官的泛化能力。