While there have been significant strides in dexterous manipulation, most of
it is limited to benchmark tasks like in-hand reorientation which are of
limited utility in the real world. The main benefit of dexterous hands over
two-fingered ones is their ability to pickup tools and other objects (including
thin ones) and grasp them firmly to apply force. However, this task requires
both a complex understanding of functional affordances as well as precise
low-level control. While prior work obtains affordances from human data this
approach doesn't scale to low-level control. Similarly, simulation training
cannot give the robot an understanding of real-world semantics. In this paper,
we aim to combine the best of both worlds to accomplish functional grasping for
in-the-wild objects. We use a modular approach. First, affordances are obtained
by matching corresponding regions of different objects and then a low-level
policy trained in sim is run to grasp it. We propose a novel application of
eigengrasps to reduce the search space of RL using a small amount of human data
and find that it leads to more stable and physically realistic motion. We find
that eigengrasp action space beats baselines in simulation and outperforms
hardcoded grasping in real and matches or outperforms a trained human
teleoperator. Results visualizations and videos at this https URL

本文旨在结合两种方法，实现对自然环境中物体的功能性抓取。首先，通过匹配不同物体的相应区域来获取各物体的可负担性，然后在模拟中运行一个经过训练的低级策略来抓取物体。我们提出了一种新颖的使用特征抓取来减少强化学习搜索空间的应用，使用少量人类数据实现更稳定和物理上逼真的运动。实验结果可在该 URL 中获得可视化和视频。

灵巧功能抓握

Dexterous Functional Grasping

We observe that pre-trained large language models (LLMs) are capable of
autoregressively completing complex token sequences -- from arbitrary ones
procedurally generated by probabilistic context-free grammars (PCFG), to more
rich spatial patterns found in the Abstract Reasoning Corpus (ARC), a general
AI benchmark, prompted in the style of ASCII art. Surprisingly, pattern
completion proficiency can be partially retained even when the sequences are
expressed using tokens randomly sampled from the vocabulary. These results
suggest that without any additional training, LLMs can serve as general
sequence modelers, driven by in-context learning. In this work, we investigate
how these zero-shot capabilities may be applied to problems in robotics -- from
extrapolating sequences of numbers that represent states over time to complete
simple motions, to least-to-most prompting of reward-conditioned trajectories
that can discover and represent closed-loop policies (e.g., a stabilizing
controller for CartPole). While difficult to deploy today for real systems due
to latency, context size limitations, and compute costs, the approach of using
LLMs to drive low-level control may provide an exciting glimpse into how the
patterns among words could be transferred to actions.

使用预训练语言模型实现模式补全、序列建模、机器人控制等任务。