In this work, we aim to learn a unified vision-based policy for a
multi-fingered robot hand to manipulate different objects in diverse poses.
Though prior work has demonstrated that human videos can benefit policy
learning, performance improvement has been limited by physically implausible
trajectories extracted from videos. Moreover, reliance on privileged object
information such as ground-truth object states further limits the applicability
in realistic scenarios. To address these limitations, we propose a new
framework ViViDex to improve vision-based policy learning from human videos. It
first uses reinforcement learning with trajectory guided rewards to train
state-based policies for each video, obtaining both visually natural and
physically plausible trajectories from the video. We then rollout successful
episodes from state-based policies and train a unified visual policy without
using any privileged information. A coordinate transformation method is
proposed to significantly boost the performance. We evaluate our method on
three dexterous manipulation tasks and demonstrate a large improvement over
state-of-the-art algorithms.

我们的研究旨在通过学习一种统一的基于视觉的策略，使多指机械手能够在各种姿势下操纵不同的物体。我们提出了一种新的框架 ViViDex，通过使用强化学习和轨迹引导奖励训练每个视频的基于状态的策略，从而获得视觉自然且物理可行的轨迹，并在不使用任何特权信息的情况下训练出一个统一的视觉策略，该方法通过坐标转换方法显著提升了性能，在三个灵巧操纵任务上展示出了较大的改进。

ViViDex: 从人类视频中学习基于视觉的灵巧操纵

ViViDex: Learning Vision-based Dexterous Manipulation from Human Videos

Hierarchical policies that combine language and low-level control have been
shown to perform impressively long-horizon robotic tasks, by leveraging either
zero-shot high-level planners like pretrained language and vision-language
models (LLMs/VLMs) or models trained on annotated robotic demonstrations.
However, for complex and dexterous skills, attaining high success rates on
long-horizon tasks still represents a major challenge -- the longer the task
is, the more likely it is that some stage will fail. Can humans help the robot
to continuously improve its long-horizon task performance through intuitive and
natural feedback? In this paper, we make the following observation: high-level
policies that index into sufficiently rich and expressive low-level
language-conditioned skills can be readily supervised with human feedback in
the form of language corrections. We show that even fine-grained corrections,
such as small movements ("move a bit to the left"), can be effectively
incorporated into high-level policies, and that such corrections can be readily
obtained from humans observing the robot and making occasional suggestions.
This framework enables robots not only to rapidly adapt to real-time language
feedback, but also incorporate this feedback into an iterative training scheme
that improves the high-level policy's ability to correct errors in both
low-level execution and high-level decision-making purely from verbal feedback.
Our evaluation on real hardware shows that this leads to significant
performance improvement in long-horizon, dexterous manipulation tasks without
the need for any additional teleoperation. Videos and code are available at
this https URL

通过语言反馈不断改进高层策略，使机器人在复杂的、需要长程规划的任务中表现得更加出色，无需额外的远程操作。

喊叫您的机器人：从语言纠正中实时改进

Yell At Your Robot: Improving On-the-Fly from Language Corrections

Human dexterity is a hallmark of motor control. Our hands can rapidly
synthesize new behaviors despite the complexity (multi-articular and
multi-joints, with 23 joints controlled by more than 40 muscles) of
musculoskeletal sensory-motor circuits. In this work, we take inspiration from
how human dexterity builds on a diversity of prior experiences, instead of
being acquired through a single task. Motivated by this observation, we set out
to develop agents that can build upon their previous experience to quickly
acquire new (previously unattainable) behaviors. Specifically, our approach
leverages multi-task learning to implicitly capture task-agnostic behavioral
priors (MyoDex) for human-like dexterity, using a physiologically realistic
human hand model - MyoHand. We demonstrate MyoDex's effectiveness in few-shot
generalization as well as positive transfer to a large repertoire of unseen
dexterous manipulation tasks. Agents leveraging MyoDex can solve approximately
3x more tasks, and 4x faster in comparison to a distillation baseline. While
prior work has synthesized single musculoskeletal control behaviors, MyoDex is
the first generalizable manipulation prior that catalyzes the learning of
dexterous physiological control across a large variety of contact-rich
behaviors. We also demonstrate the effectiveness of our paradigms beyond
musculoskeletal control towards the acquisition of dexterity in 24 DoF Adroit
Hand. Website: this https URL

利用多任务学习模型和行为先验，研究人类灵巧性和机器人物理控制在各种接触丰富的行为中的学习和操作能力。