A compelling use case of offline reinforcement learning (RL) is to obtain a
policy initialization from existing datasets, which allows efficient
fine-tuning with limited amounts of active online interaction. However, several
existing offline RL methods tend to exhibit poor online fine-tuning
performance. On the other hand, online RL methods can learn effectively through
online interaction, but struggle to incorporate offline data, which can make
them very slow in settings where exploration is challenging or pre-training is
necessary. In this paper, we devise an approach for learning an effective
initialization from offline data that also enables fast online fine-tuning
capabilities. Our approach, calibrated Q-learning (Cal-QL) accomplishes this by
learning a conservative value function initialization that underestimates the
value of the learned policy from offline data, while also being calibrated, in
the sense that the learned Q-values are at a reasonable scale. We refer to this
property as calibration, and define it formally as providing a lower bound on
the true value function of the learned policy and an upper bound on the value
of some other (suboptimal) reference policy, which may simply be the behavior
policy. We show that offline RL algorithms that learn such calibrated value
functions lead to effective online fine-tuning, enabling us to take the
benefits of offline initializations in online fine-tuning. In practice, Cal-QL
can be implemented on top of existing conservative methods for offline RL
within a one-line code change. Empirically, Cal-QL outperforms state-of-the-art
methods on 10/11 fine-tuning benchmark tasks that we study in this paper.

本文提出了一种新颖的基于离线数据初始化的强化学习 (RL) 初始值学习框架，称作 Cal-QL，该方法可以快速优化在离线数据上学习的策略，并在探索困难的情况下进行预训练，实验证明该方法在 10/11 个测试任务中超过了现有成熟方法。