Multimodal task specification is essential for enhanced robotic performance, where \textit{Cross-modality Alignment} enables the robot to holistically understand complex task instructions. Directly annotating multimodal instructions for model training proves impractical, due to the sparsity of paired multimodal data. In this study, we demonstrate that by leveraging unimodal instructions abundant in real data, we can effectively teach robots to learn multimodal task specifications. First, we endow the robot with strong \textit{Cross-modality Alignment} capabilities, by pretraining a robotic multimodal encoder using extensive out-of-domain data. Then, we employ two Collapse and Corrupt operations to further bridge the remaining modality gap in the learned multimodal representation. This approach projects different modalities of identical task goal as interchangeable representations, thus enabling accurate robotic operations within a well-aligned multimodal latent space. Evaluation across more than 130 tasks and 4000 evaluations on both simulated LIBERO benchmark and real robot platforms showcases the superior capabilities of our proposed framework, demonstrating significant advantage in overcoming data constraints in robotic learning. Website: zh1hao.wang/Robo_MUTUAL

本研究解决了机器人在多模态任务规范中的数据稀缺问题，采用单模态指令进行学习。通过 pretraining 强化跨模态对齐能力，并使用 Collapse 和 Corrupt 操作，成功弥补了多模态表示中的差距，使机器人能够在对齐的多模态潜在空间中精确执行任务。评估表明，该框架在超过130个任务和4000次评估中表现优越，有助于缓解机器人学习中的数据约束。

Robo-MUTUAL：通过单模态学习实现的机器人多模态任务规范