We show that offline actor-critic reinforcement learning can scale to large
models - such as transformers - and follows similar scaling laws as supervised
learning. We find that offline actor-critic algorithms can outperform strong,
supervised, behavioral cloning baselines for multi-task training on a large
dataset containing both sub-optimal and expert behavior on 132 continuous
control tasks. We introduce a Perceiver-based actor-critic model and elucidate
the key model features needed to make offline RL work with self- and
cross-attention modules. Overall, we find that: i) simple offline actor critic
algorithms are a natural choice for gradually moving away from the currently
predominant paradigm of behavioral cloning, and ii) via offline RL it is
possible to learn multi-task policies that master many domains simultaneously,
including real robotics tasks, from sub-optimal demonstrations or
self-generated data.

离线演员 - 评论家强化学习能够扩展到大规模模型（如变压器）并遵循监督学习的相似扩展规律。我们发现，离线演员 - 评论家算法在 132 个连续控制任务的大型数据集上的多任务训练中，能够优于强大的监督式行为克隆基线。我们引入了一种基于感知机的离线演员 - 评论家模型，并阐明了使离线强化学习与自注意力和跨注意力模块配合工作的关键模型特征。总体而言，我们发现：i）简单的离线演员评论家算法是逐渐摆脱当前主导的行为克隆范式的自然选择，ii）通过离线强化学习，可以从次优示范或自生成数据中学习掌握多个领域的多任务策略，包括真实的机器人任务。