This document serves as a position paper that outlines the authors' vision
for a potential pathway towards generalist robots. The purpose of this document
is to share the excitement of the authors with the community and highlight a
promising research direction in robotics and AI. The authors believe the
proposed paradigm is a feasible path towards accomplishing the long-standing
goal of robotics research: deploying robots, or embodied AI agents more
broadly, in various non-factory real-world settings to perform diverse tasks.
This document presents a specific idea for mining knowledge in the latest
large-scale foundation models for robotics research. Instead of directly
adapting these models or using them to guide low-level policy learning, it
advocates for using them to generate diversified tasks and scenes at scale,
thereby scaling up low-level skill learning and ultimately leading to a
foundation model for robotics that empowers generalist robots. The authors are
actively pursuing this direction, but in the meantime, they recognize that the
ambitious goal of building generalist robots with large-scale policy training
demands significant resources such as computing power and hardware, and
research groups in academia alone may face severe resource constraints in
implementing the entire vision. Therefore, the authors believe sharing their
thoughts at this early stage could foster discussions, attract interest towards
the proposed pathway and related topics from industry groups, and potentially
spur significant technical advancements in the field.

该文呈现了一种用于采矿最新大规模基础模型中的知识的具体想法，其目的是将低级技能学习扩展至规模化，从而实现授予通才机器人的基础模型，并享有在各种现实场景中执行多种任务的能力。

通向通用型机器人的基础模型：通过自动化任务和场景生成实现大规模的多样化技能学习

Towards A Foundation Model for Generalist Robots: Diverse Skill Learning  at Scale via Automated Task and Scene Generation

Recently, synthetic data generation and realistic rendering has advanced
tasks like target tracking and human pose estimation. Simulations for most
robotics applications are obtained in (semi)static environments, with specific
sensors and low visual fidelity. To solve this, we present a fully customizable
framework for generating realistic animated dynamic environments (GRADE) for
robotics research, first introduced in [1]. GRADE supports full simulation
control, ROS integration, realistic physics, while being in an engine that
produces high visual fidelity images and ground truth data. We use GRADE to
generate a dataset focused on indoor dynamic scenes with people and flying
objects. Using this, we evaluate the performance of YOLO and Mask R-CNN on the
tasks of segmenting and detecting people. Our results provide evidence that
using data generated with GRADE can improve the model performance when used for
a pre-training step. We also show that, even training using only synthetic
data, can generalize well to real-world images in the same application domain
such as the ones from the TUM-RGBD dataset. The code, results, trained models,
and the generated data are provided as open-source at
this https URL

提出了用于机器人研究的可定制的、真实动态环境生成框架 (GRADE)，利用该框架生成数据集用于检测和分割人物，并表明使用该数据集进行预训练能够提高模型性能

利用 GRADE 生成的合成数据进行学习

Learning from synthetic data generated with GRADE

We introduce BenchBot, a novel software suite for benchmarking the
performance of robotics research across both photorealistic 3D simulations and
real robot platforms. BenchBot provides a simple interface to the sensorimotor
capabilities of a robot when solving robotics research problems; an interface
that is consistent regardless of whether the target platform is simulated or a
real robot. In this paper we outline the BenchBot system architecture, and
explore the parallels between its user-centric design and an ideal research
development process devoid of tangential robot engineering challenges. The
paper describes the research benefits of using the BenchBot system, including:
enhanced capacity to focus solely on research problems, direct quantitative
feedback to inform research development, tools for deriving comprehensive
performance characteristics, and submission formats which promote sharability
and repeatability of research outcomes. BenchBot is publicly available
(this http URL), and we encourage its use in the research community for
comprehensively evaluating the simulated and real world performance of novel
robotic algorithms.

本研究介绍了一种名为 BenchBot 的软件套件，该套件旨在通过在逼真的 3D 模拟和现实机器人平台上进行评估来评估机器人研究的表现。BenchBot 为使用者提供了一种接口，用于解决机器人研究问题时机器人的机感能力。同时我们说明了使用 BenchBot 系统的研究优势，包括：增强了专注于研究问题、直接量化反馈信息以促进研究发展、工具用于派生全面的性能特征、以及促进研究结果的可共享和可重复性的提交格式。