In this work, we aim to learn dexterous manipulation of deformable objects
using multi-fingered hands. Reinforcement learning approaches for dexterous
rigid object manipulation would struggle in this setting due to the complexity
of physics interaction with deformable objects. At the same time, previous
trajectory optimization approaches with differentiable physics for deformable
manipulation would suffer from local optima caused by the explosion of contact
modes from hand-object interactions. To address these challenges, we propose
DexDeform, a principled framework that abstracts dexterous manipulation skills
from human demonstration and refines the learned skills with differentiable
physics. Concretely, we first collect a small set of human demonstrations using
teleoperation. And we then train a skill model using demonstrations for
planning over action abstractions in imagination. To explore the goal space, we
further apply augmentations to the existing deformable shapes in demonstrations
and use a gradient optimizer to refine the actions planned by the skill model.
Finally, we adopt the refined trajectories as new demonstrations for finetuning
the skill model. To evaluate the effectiveness of our approach, we introduce a
suite of six challenging dexterous deformable object manipulation tasks.
Compared with baselines, DexDeform is able to better explore and generalize
across novel goals unseen in the initial human demonstrations.

本文提出了 DexDeform，这是一种从人类示范中抽象出熟练操作技能并通过可微分物理学进行改进的原则性框架，该框架旨在学习多指手指对可变形物体的熟练操作，并通过新颖的探索策略以及基于想象的行动计划来提高操作的成功率。