We propose VRL3, a powerful data-driven framework with a simple design for
solving challenging visual deep reinforcement learning (DRL) tasks. We analyze
a number of major obstacles in taking a data-driven approach, and present a
suite of design principles, novel findings, and critical insights about
data-driven visual DRL. Our framework has three stages: in stage 1, we leverage
non-RL datasets (e.g. ImageNet) to learn task-agnostic visual representations;
in stage 2, we use offline RL data (e.g. a limited number of expert
demonstrations) to convert the task-agnostic representations into more powerful
task-specific representations; in stage 3, we fine-tune the agent with online
RL. On a set of challenging hand manipulation tasks with sparse reward and
realistic visual inputs, compared to the previous SOTA, VRL3 achieves an
average of 780% better sample efficiency. And on the hardest task, VRL3 is
1220% more sample efficient (2440% when using a wider encoder) and solves the
task with only 10% of the computation. These significant results clearly
demonstrate the great potential of data-driven deep reinforcement learning.

提出了 VRL3 这一数据驱动框架，在视觉深度强化学习中取得了极高的效率，并且在手部操作任务中的样本效率比先前最先进的方法高出了 780％。VRL3 在最困难的任务中样本效率提高了 1220％（使用更宽的编码器达到 2440％）并且只需 10％的计算量就能解决该任务，这些结果清楚地证明了数据驱动深度强化学习的巨大潜力。