Aiming to replicate human-like dexterity, perceptual experiences, and motion
patterns, we explore learning from human demonstrations using a bimanual system
with multifingered hands and visuotactile data. Two significant challenges
exist: the lack of an affordable and accessible teleoperation system suitable
for a dual-arm setup with multifingered hands, and the scarcity of
multifingered hand hardware equipped with touch sensing. To tackle the first
challenge, we develop HATO, a low-cost hands-arms teleoperation system that
leverages off-the-shelf electronics, complemented with a software suite that
enables efficient data collection; the comprehensive software suite also
supports multimodal data processing, scalable policy learning, and smooth
policy deployment. To tackle the latter challenge, we introduce a novel
hardware adaptation by repurposing two prosthetic hands equipped with touch
sensors for research. Using visuotactile data collected from our system, we
learn skills to complete long-horizon, high-precision tasks which are difficult
to achieve without multifingered dexterity and touch feedback. Furthermore, we
empirically investigate the effects of dataset size, sensing modality, and
visual input preprocessing on policy learning. Our results mark a promising
step forward in bimanual multifingered manipulation from visuotactile data.
Videos, code, and datasets can be found at this https URL .

通过学习人类示范，我们利用双臂系统和视触数据探索实现类似人类敏捷性、感知体验和运动模式的目标。我们开发了一种低成本的手臂远程操作系统并重新利用配备触觉传感器的义肢手，通过采集到的视触数据学习完成长周期、高精度任务。我们的结果在双臂多指操作和视触数据方面迈出了有希望的一步。

使用两只多指手学习视触技能

Learning Visuotactile Skills with Two Multifingered Hands

Folding garments reliably and efficiently is a long standing challenge in
robotic manipulation due to the complex dynamics and high dimensional
configuration space of garments. An intuitive approach is to initially
manipulate the garment to a canonical smooth configuration before folding. In
this work, we develop SpeedFolding, a reliable and efficient bimanual system,
which given user-defined instructions as folding lines, manipulates an
initially crumpled garment to (1) a smoothed and (2) a folded configuration.
Our primary contribution is a novel neural network architecture that is able to
predict pairs of gripper poses to parameterize a diverse set of bimanual action
primitives. After learning from 4300 human-annotated and self-supervised
actions, the robot is able to fold garments from a random initial configuration
in under 120s on average with a success rate of 93%. Real-world experiments
show that the system is able to generalize to unseen garments of different
color, shape, and stiffness. While prior work achieved 3-6 Folds Per Hour
(FPH), SpeedFolding achieves 30-40 FPH.

本文描述了一个名为 SpeedFolding 的双手系统，使用一种新颖的神经网络体系结构来预测钳爪姿势的配对，实现对衣物的快速折叠，可在平均不到 120 秒的时间内折叠多达 30-40 次，并成功适应不同颜色、形状和硬度的衣物。