In this paper, we introduce RealDex, a pioneering dataset capturing authentic
dexterous hand grasping motions infused with human behavioral patterns,
enriched by multi-view and multimodal visual data. Utilizing a teleoperation
system, we seamlessly synchronize human-robot hand poses in real time. This
collection of human-like motions is crucial for training dexterous hands to
mimic human movements more naturally and precisely. RealDex holds immense
promise in advancing humanoid robot for automated perception, cognition, and
manipulation in real-world scenarios. Moreover, we introduce a cutting-edge
dexterous grasping motion generation framework, which aligns with human
experience and enhances real-world applicability through effectively utilizing
Multimodal Large Language Models. Extensive experiments have demonstrated the
superior performance of our method on RealDex and other open datasets. The
complete dataset and code will be made available upon the publication of this
work.

在这篇论文中，我们介绍了 RealDex—— 一个捕捉了真实灵巧手部抓握动作的先驱数据集，其中融入了人类行为模式，丰富了多视角和多模态的视觉数据。使用远程操作系统，我们实时无缝地同步人 - 机器手的姿态。这个类似人类运动的集合对于训练灵巧手更自然、更准确地模仿人类动作至关重要。RealDex 在推动人形机器人在现实世界场景中的自动感知、认知和操作方面具有巨大潜力。此外，我们介绍了一种先进的灵巧抓握动作生成框架，通过有效利用多模态的大型语言模型与人类经验相吻合，增强了其在现实世界的应用能力。大量实验证明了我们的方法在 RealDex 和其他公开数据集上的出色性能。完整的数据集和代码将在本文发表后提供。