Learning a generalist embodied agent capable of completing multiple tasks
poses challenges, primarily stemming from the scarcity of action-labeled
robotic datasets. In contrast, a vast amount of human videos exist, capturing
intricate tasks and interactions with the physical world. Promising prospects
arise for utilizing actionless human videos for pre-training and transferring
the knowledge to facilitate robot policy learning through limited robot
demonstrations. In this paper, we introduce a novel framework that leverages a
unified discrete diffusion to combine generative pre-training on human videos
and policy fine-tuning on a small number of action-labeled robot videos. We
start by compressing both human and robot videos into unified video tokens. In
the pre-training stage, we employ a discrete diffusion model with a
mask-and-replace diffusion strategy to predict future video tokens in the
latent space. In the fine-tuning stage, we harness the imagined future videos
to guide low-level action learning trained on a limited set of robot data.
Experiments demonstrate that our method generates high-fidelity future videos
for planning and enhances the fine-tuned policies compared to previous
state-of-the-art approaches with superior generalization ability. Our project
website is available at this https URL

利用人类视频进行预训练，通过有限的机器人演示来转移知识，实现机器人策略学习的通用性体现。