With the advent of large language models and large-scale robotic datasets, there has been tremendous progress in high-level decision-making for object manipulation. These generic models are able to interpret complex tasks using language commands, but they often have difficulties generalizing to out-of-distribution objects due to the inability of low-level action primitives. In contrast, existing task-specific models excel in low-level manipulation of unknown objects, but only work for a single type of action. To bridge this gap, we present M2T2, a single model that supplies different types of low-level actions that work robustly on arbitrary objects in cluttered scenes. M2T2 is a transformer model which reasons about contact points and predicts valid gripper poses for different action modes given a raw point cloud of the scene. Trained on a large-scale synthetic dataset with 128K scenes, M2T2 achieves zero-shot sim2real transfer on the real robot, outperforming the baseline system with state-of-the-art task-specific models by about 19% in overall performance and 37.5% in challenging scenes where the object needs to be re-oriented for collision-free placement. M2T2 also achieves state-of-the-art results on a subset of language conditioned tasks in RLBench. Videos of robot experiments on unseen objects in both real world and simulation are available on our project website https://m2-t2.github.io.

M2T2是一种使用Transformer模型的单一模型，它能够供应不同类型的低级动作，在杂乱的场景中对任意对象进行稳定工作。训练于128K个场景的大规模合成数据集，M2T2在真实机器人上实现了零样本仿真到现实的转换，在整体性能上超过了基准系统和最先进的任务特定模型约19％，在需要重新定向以进行无碰放置的挑战性场景中超过了37.5％。M2T2还在RLBench的一部分以语言为条件的任务上实现了最先进的结果。项目网站提供了真实世界和仿真环境中未知对象的机器人实验视频。

M2T2：多任务遮罩变形器用于以对象为中心的拾放