Imitation learning from human hand motion data presents a promising avenue
for imbuing robots with human-like dexterity in real-world manipulation tasks.
Despite this potential, substantial challenges persist, particularly with the
portability of existing hand motion capture (mocap) systems and the difficulty
of translating mocap data into effective control policies. To tackle these
issues, we introduce DexCap, a portable hand motion capture system, alongside
DexIL, a novel imitation algorithm for training dexterous robot skills directly
from human hand mocap data. DexCap offers precise, occlusion-resistant tracking
of wrist and finger motions based on SLAM and electromagnetic field together
with 3D observations of the environment. Utilizing this rich dataset, DexIL
employs inverse kinematics and point cloud-based imitation learning to
replicate human actions with robot hands. Beyond learning from human motion,
DexCap also offers an optional human-in-the-loop correction mechanism to refine
and further improve robot performance. Through extensive evaluation across six
dexterous manipulation tasks, our approach not only demonstrates superior
performance but also showcases the system's capability to effectively learn
from in-the-wild mocap data, paving the way for future data collection methods
for dexterous manipulation. More details can be found at
this https URL

通过 DexCap 便携式手部动作捕捉系统和 DexIL 创新的模仿算法，实现从人类手部动作数据中直接训练机器人掌握人类类似的灵巧操作技能，并通过 6 个巧妙操作任务的广泛评估，证明了该方法在性能和对野外动作数据的学习效果上的优越性，为灵巧操作的未来数据收集方法铺平了道路。

DexCap：灵巧操控运动捕捉数据采集系统的可伸缩和可移植性

DexCap: Scalable and Portable Mocap Data Collection System for Dexterous  Manipulation

Hand motion capture has been an active research topic in recent years,
following the success of full-body pose tracking. Despite similarities, hand
tracking proves to be more challenging, characterized by a higher
dimensionality, severe occlusions and self-similarity between fingers. For this
reason, most approaches rely on strong assumptions, like hands in isolation or
expensive multi-camera systems, that limit the practical use. In this work, we
propose a framework for hand tracking that can capture the motion of two
interacting hands using only a single, inexpensive RGB-D camera. Our approach
combines a generative model with collision detection and discriminatively
learned salient points. We quantitatively evaluate our approach on 14 new
sequences with challenging interactions.

手势运动捕捉的关键挑战在于质量和价格，该研究发现单个低成本 RGB-D 相机与生成模型，碰撞检测和重要点的结合可以捕捉运动中的两个交互手。

使用 RGB-D 传感器捕捉手部动作，将生成模型与显著点融合

Capturing Hand Motion with an RGB-D Sensor, Fusing a Generative Model  with Salient Points

Hand motion capture is a popular research field, recently gaining more
attention due to the ubiquity of RGB-D sensors. However, even most recent
approaches focus on the case of a single isolated hand. In this work, we focus
on hands that interact with other hands or objects and present a framework that
successfully captures motion in such interaction scenarios for both rigid and
articulated objects. Our framework combines a generative model with
discriminatively trained salient points to achieve a low tracking error and
with collision detection and physics simulation to achieve physically plausible
estimates even in case of occlusions and missing visual data. Since all
components are unified in a single objective function which is almost
everywhere differentiable, it can be optimized with standard optimization
techniques. Our approach works for monocular RGB-D sequences as well as setups
with multiple synchronized RGB cameras. For a qualitative and quantitative
evaluation, we captured 29 sequences with a large variety of interactions and
up to 150 degrees of freedom.

这篇文章提出了一种手势捕捉框架，其结合了生成模型和歧视性训练显著点，以实现对物体和其他手势之间互动场景的精确捕捉，并使用碰撞检测和物理模拟实现了在遮挡和缺失数据情况下的准确估计，能适用于单目 RGB-D 序列和同步多 RGB 摄像头的设置。