Learning to solve tasks from a sparse reward signal is a major challenge for standard reinforcement learning (RL) algorithms. However, in the real world, agents rarely need to solve sparse reward tasks entirely from scratch. More often, we might possess prior experience to draw on that provides considerable guidance about which actions and outcomes are possible in the world, which we can use to explore more effectively for new tasks. In this work, we study how prior data without reward labels may be used to guide and accelerate exploration for an agent solving a new sparse reward task. We propose a simple approach that learns a reward model from online experience, labels the unlabeled prior data with optimistic rewards, and then uses it concurrently alongside the online data for downstream policy and critic optimization. This general formula leads to rapid exploration in several challenging sparse-reward domains where tabula rasa exploration is insufficient, including the AntMaze domain, Adroit hand manipulation domain, and a visual simulated robotic manipulation domain. Our results highlight the ease of incorporating unlabeled prior data into existing online RL algorithms, and the (perhaps surprising) effectiveness of doing so.

从稀疏奖励信号学习解决任务是标准强化学习算法的一个重大挑战。然而，在现实世界中，代理很少需要完全从头开始解决稀疏奖励任务。本研究探讨了如何利用没有奖励标签的先前数据来指导和加速代理解决新的稀疏奖励任务，并提出了一种简单的方法，通过在线经验学习奖励模型，使用乐观奖励对未标记的先前数据进行标记，并与在线数据同时用于下游策略和评论家优化。研究结果表明，将未标记的先前数据纳入现有的在线强化学习算法非常容易，并且这样做的效果出乎意料地好。

无标签先验数据加速探索