Offline reinforcement learning (RL) enables learning control policies by
utilizing only prior experience, without any online interaction. This can allow
robots to acquire generalizable skills from large and diverse datasets, without
any costly or unsafe online data collection. Despite recent algorithmic
advances in offline RL, applying these methods to real-world problems has
proven challenging. Although offline RL methods can learn from prior data,
there is no clear and well-understood process for making various design
choices, from model architecture to algorithm hyperparameters, without actually
evaluating the learned policies online. In this paper, our aim is to develop a
practical workflow for using offline RL analogous to the relatively
well-understood workflows for supervised learning problems. To this end, we
devise a set of metrics and conditions that can be tracked over the course of
offline training, and can inform the practitioner about how the algorithm and
model architecture should be adjusted to improve final performance. Our
workflow is derived from a conceptual understanding of the behavior of
conservative offline RL algorithms and cross-validation in supervised learning.
We demonstrate the efficacy of this workflow in producing effective policies
without any online tuning, both in several simulated robotic learning scenarios
and for three tasks on two distinct real robots, focusing on learning
manipulation skills with raw image observations with sparse binary rewards.
Explanatory video and additional results can be found at
sites.google.com/view/offline-rl-workflow

本文提出了一种实用的线下强化学习工作流程，类似于监督学习问题的工作流程，并演示了在几个模拟机器人学习场景和两个不同真实机器人上的三个任务中，使用此工作流程在没有在线调整的情况下生成有效策略的有效性。