Offline Reinforcement Learning (RL) aims to learn policies from previously
collected datasets without exploring the environment. Directly applying
off-policy algorithms to offline RL usually fails due to the extrapolation
error caused by the out-of-distribution (OOD) actions. Previous methods tackle
such problem by penalizing the Q-values of OOD actions or constraining the
trained policy to be close to the behavior policy. Nevertheless, such methods
typically prevent the generalization of value functions beyond the offline data
and also lack precise characterization of OOD data. In this paper, we propose
Pessimistic Bootstrapping for offline RL (PBRL), a purely uncertainty-driven
offline algorithm without explicit policy constraints. Specifically, PBRL
conducts uncertainty quantification via the disagreement of bootstrapped
Q-functions, and performs pessimistic updates by penalizing the value function
based on the estimated uncertainty. To tackle the extrapolating error, we
further propose a novel OOD sampling method. We show that such OOD sampling and
pessimistic bootstrapping yields provable uncertainty quantifier in linear
MDPs, thus providing the theoretical underpinning for PBRL. Extensive
experiments on D4RL benchmark show that PBRL has better performance compared to
the state-of-the-art algorithms.

本文提出了一种基于纯不确定性驱动的离线策略学习算法 - 悲观引导离线学习 (PBRL)，它通过引入一种 Q 函数的不确定度来量化不确定性，并以此进行悲观更新，以解决离线学习中由行为分布外数据所产生的外推误差问题。实验证明，相比现有算法，PBRL 具有更好的性能表现。