Learning effective reinforcement learning (RL) policies to solve real-world
complex tasks can be quite challenging without a high-fidelity simulation
environment. In most cases, we are only given imperfect simulators with
simplified dynamics, which inevitably lead to severe sim-to-real gaps in RL
policy learning. The recently emerged field of offline RL provides another
possibility to learn policies directly from pre-collected historical data.
However, to achieve reasonable performance, existing offline RL algorithms need
impractically large offline data with sufficient state-action space coverage
for training. This brings up a new question: is it possible to combine learning
from limited real data in offline RL and unrestricted exploration through
imperfect simulators in online RL to address the drawbacks of both approaches?
In this study, we propose the Dynamics-Aware Hybrid Offline-and-Online
Reinforcement Learning (H2O) framework to provide an affirmative answer to this
question. H2O introduces a dynamics-aware policy evaluation scheme, which
adaptively penalizes the Q function learning on simulated state-action pairs
with large dynamics gaps, while also simultaneously allowing learning from a
fixed real-world dataset. Through extensive simulation and real-world tasks, as
well as theoretical analysis, we demonstrate the superior performance of H2O
against other cross-domain online and offline RL algorithms. H2O provides a
brand new hybrid offline-and-online RL paradigm, which can potentially shed
light on future RL algorithm design for solving practical real-world tasks.

本研究提出了一个新的混合离线 - 在线强化学习范式，通过使用有限的真实数据和不受限制的模拟器探索来解决两种方法的缺陷，并在通过广泛的仿真和实际任务以及理论分析中证明了 H2O 优于其他跨域在线和离线强化学习算法，从而为解决实际的复杂任务提供了全新的思路。