The ability to leverage heterogeneous robotic experience from different
robots and tasks to quickly master novel skills and embodiments has the
potential to transform robot learning. Inspired by recent advances in
foundation models for vision and language, we propose a foundation agent for
robotic manipulation. This agent, named RoboCat, is a visual goal-conditioned
decision transformer capable of consuming multi-embodiment action-labelled
visual experience. This data spans a large repertoire of motor control skills
from simulated and real robotic arms with varying sets of observations and
actions. With RoboCat, we demonstrate the ability to generalise to new tasks
and robots, both zero-shot as well as through adaptation using only 100--1000
examples for the target task. We also show how a trained model itself can be
used to generate data for subsequent training iterations, thus providing a
basic building block for an autonomous improvement loop. We investigate the
agent's capabilities, with large-scale evaluations both in simulation and on
three different real robot embodiments. We find that as we grow and diversify
its training data, RoboCat not only shows signs of cross-task transfer, but
also becomes more efficient at adapting to new tasks.

提出了一个基于视觉目标调节的决策转换器，命名为 RoboCat，它能够消耗多体验行为标记的视觉经验，用于机器人操作的基础模型，并展示了模型对新任务和机器人的泛化能力，并可用于生成下一轮训练集的数据。

RoboCat：面向机器人操控的自我改进基础代理

RoboCat: A Self-Improving Foundation Agent for Robotic Manipulation

Inspired by progress in large-scale language modeling, we apply a similar
approach towards building a single generalist agent beyond the realm of text
outputs. The agent, which we refer to as Gato, works as a multi-modal,
multi-task, multi-embodiment generalist policy. The same network with the same
weights can play Atari, caption images, chat, stack blocks with a real robot
arm and much more, deciding based on its context whether to output text, joint
torques, button presses, or other tokens. In this report we describe the model
and the data, and document the current capabilities of Gato.

本研究基于大规模语言模型的进展，采用类似方法构建了一个跨越文本输出领域之外的单一通用智能体，称为 Gato，它作为一个多模态、多任务、多具象化的通用策略工作。相同的网络和相同的权重可以同时用于打 Atari 游戏、描述图像、聊天、使用真实机器人臂堆叠积木等任务，根据情境决定是否输出文本、关节扭矩、按键按下或其他词汇，并在本文中描述该模型和数据，并记录 Gato 的当前能力。