Recent advancements in Artificial Intelligence (AI) have largely been
propelled by scaling. In Robotics, scaling is hindered by the lack of access to
massive robot datasets. We advocate using realistic physical simulation as a
means to scale environments, tasks, and datasets for robot learning methods. We
present RoboCasa, a large-scale simulation framework for training generalist
robots in everyday environments. RoboCasa features realistic and diverse scenes
focusing on kitchen environments. We provide thousands of 3D assets across over
150 object categories and dozens of interactable furniture and appliances. We
enrich the realism and diversity of our simulation with generative AI tools,
such as object assets from text-to-3D models and environment textures from
text-to-image models. We design a set of 100 tasks for systematic evaluation,
including composite tasks generated by the guidance of large language models.
To facilitate learning, we provide high-quality human demonstrations and
integrate automated trajectory generation methods to substantially enlarge our
datasets with minimal human burden. Our experiments show a clear scaling trend
in using synthetically generated robot data for large-scale imitation learning
and show great promise in harnessing simulation data in real-world tasks.
Videos and open-source code are available at this https URL

人工智能的近期发展大多受到规模化的推动。在机器人学中，规模化受到获取大规模机器人数据集的限制。我们提倡使用逼真的物理仿真作为扩展环境、任务和数据集以用于机器人学习方法。我们提出了一个名为 RoboCasa 的大规模仿真框架，用于在日常环境中训练通用型机器人。RoboCasa 以厨房环境为重点，具备逼真和多样化的场景。我们提供了数千个 3D 物体，涵盖 150 多个物体类别和数十个可交互的家具和电器。我们利用生成式人工智能工具来丰富仿真的逼真度和多样性，如使用文本生成 3D 模型的物体和使用文本生成环境纹理的模型。我们设计了一套包含 100 个任务的系统评估，其中包括由大型语言模型引导生成的复合任务。为了便于学习，我们提供高质量的人类演示，并整合自动化轨迹生成方法，以最小化人力负担大幅扩展数据集。我们的实验显示，使用合成生成的机器人数据进行大规模的模仿学习呈现出明显的规模化趋势，并在利用仿真数据进行真实任务中表现出巨大潜力。视频和开源代码可在此 https 网址上获取。

RoboCasa: 为通用机器人实现大规模模拟日常任务

RoboCasa: Large-Scale Simulation of Everyday Tasks for Generalist Robots

Recent studies in Vision-and-Language Navigation (VLN) train RL agents to
execute natural-language navigation instructions in photorealistic
environments, as a step towards robots that can follow human instructions.
However, given the scarcity of human instruction data and limited diversity in
the training environments, these agents still struggle with complex language
grounding and spatial language understanding. Pretraining on large text and
image-text datasets from the web has been extensively explored but the
improvements are limited. We investigate large-scale augmentation with
synthetic instructions. We take 500+ indoor environments captured in
densely-sampled 360 degree panoramas, construct navigation trajectories through
these panoramas, and generate a visually-grounded instruction for each
trajectory using Marky, a high-quality multilingual navigation instruction
generator. We also synthesize image observations from novel viewpoints using an
image-to-image GAN. The resulting dataset of 4.2M instruction-trajectory pairs
is two orders of magnitude larger than existing human-annotated datasets, and
contains a wider variety of environments and viewpoints. To efficiently
leverage data at this scale, we train a simple transformer agent with imitation
learning. On the challenging RxR dataset, our approach outperforms all existing
RL agents, improving the state-of-the-art NDTW from 71.1 to 79.1 in seen
environments, and from 64.6 to 66.8 in unseen test environments. Our work
points to a new path to improving instruction-following agents, emphasizing
large-scale imitation learning and the development of synthetic instruction
generation capabilities.

利用 360 度全景数据生成的合成语音引导大型模拟数据集，使用模仿学习的简单 Transformer 模型，本文介绍了一种新的方式 —— 合成指令及大规模模仿学习来提高代理程序。