Offline pretraining with a static dataset followed by online fine-tuning
(offline-to-online, or OtO) is a paradigm that is well matched to a real-world
RL deployment process: in few real settings would one deploy an offline policy
with no test runs and tuning. In this scenario, we aim to find the
best-performing policy within a limited budget of online interactions. Previous
work in the OtO setting has focused on correcting for bias introduced by the
policy-constraint mechanisms of offline RL algorithms. Such constraints keep
the learned policy close to the behavior policy that collected the dataset, but
this unnecessarily limits policy performance if the behavior policy is far from
optimal. Instead, we forgo policy constraints and frame OtO RL as an
exploration problem: we must maximize the benefit of the online
data-collection. We study major online RL exploration paradigms, adapting them
to work well with the OtO setting. These adapted methods contribute several
strong baselines. Also, we introduce an algorithm for planning to go out of
distribution (PTGOOD), which targets online exploration in relatively
high-reward regions of the state-action space unlikely to be visited by the
behavior policy. By leveraging concepts from the Conditional Entropy
Bottleneck, PTGOOD encourages data collected online to provide new information
relevant to improving the final deployment policy. In that way the limited
interaction budget is used effectively. We show that PTGOOD significantly
improves agent returns during online fine-tuning and finds the optimal policy
in as few as 10k online steps in Walker and in as few as 50k in complex control
tasks like Humanoid. Also, we find that PTGOOD avoids the suboptimal policy
convergence that many of our baselines exhibit in several environments.

离线预训练与在线微调（offline-to-online 或 OtO）是与实际强化学习部署过程相匹配的范式，我们研究在线强化学习开拓问题的主要方法，其中 PTGOOD 算法在在线微调中显著提高了智能体回报，并且在少于 10k 的在线步骤中找到了最优策略。