Multi-step dexterous manipulation is a fundamental skill in household scenarios, yet remains an underexplored area in robotics. This paper proposes a modular approach, where each step of the manipulation process is addressed with dedicated policies based on effective modality input, rather than relying on a single end-to-end model. To demonstrate this, a dexterous robotic hand performs a manipulation task involving picking up and rotating a box. Guided by insights from neuroscience, the task is decomposed into three sub-skills, 1)reaching, 2)grasping and lifting, and 3)in-hand rotation, based on the dominant sensory modalities employed in the human brain. Each sub-skill is addressed using distinct methods from a practical perspective: a classical controller, a Vision-Language-Action model, and a reinforcement learning policy with force feedback, respectively. We tested the pipeline on a real robot to demonstrate the feasibility of our approach. The key contribution of this study lies in presenting a neuroscience-inspired, modality-driven methodology for multi-step dexterous manipulation.

本研究针对多步骤灵巧操控在机器人领域的不足，提出了一种模块化的方法，针对每一个操作步骤采用有效的模态输入制定专门策略，而非依赖单一的端到端模型。通过神经科学的启示，任务被分解为三个子技能，并在真实机器人上进行了测试，展示了研究方法的可行性，具有重要的应用潜力。

基于模态驱动的多步骤灵巧操控设计：来自神经科学的启示