Bimanual dexterous manipulation remains significant challenges in robotics due to the high DoFs of each hand and their coordination. Existing single-hand manipulation techniques often leverage human demonstrations to guide RL methods but fail to generalize to complex bimanual tasks involving multiple sub-skills. In this paper, we introduce VTAO-BiManip, a novel framework that combines visual-tactile-action pretraining with object understanding to facilitate curriculum RL to enable human-like bimanual manipulation. We improve prior learning by incorporating hand motion data, providing more effective guidance for dual-hand coordination than binary tactile feedback. Our pretraining model predicts future actions as well as object pose and size using masked multimodal inputs, facilitating cross-modal regularization. To address the multi-skill learning challenge, we introduce a two-stage curriculum RL approach to stabilize training. We evaluate our method on a bottle-cap unscrewing task, demonstrating its effectiveness in both simulated and real-world environments. Our approach achieves a success rate that surpasses existing visual-tactile pretraining methods by over 20%.

本研究解决了双手灵巧操作中的协调和高自由度（DoFs）问题，现有方法往往无法有效应对复杂的双手任务。我们提出的VTAO-BiManip框架结合了视觉、触觉和动作预训练，增强了双手协调的指导性，并通过两阶课程强化学习的方法来稳定培训。实验结果表明，该方法在瓶盖拧松任务中，在模拟和现实环境中的成功率超越了现有的视觉-触觉预训练方法超过20%。

VTAO-BiManip：具有物体理解的掩蔽视觉-触觉-动作预训练用于双手灵巧操作