We study learning optimal policies from a logged dataset, i.e., offline RL, with function approximation. Despite the efforts devoted, existing algorithms with theoretic finite-sample guarantees typically assume exploratory data coverage or strong realizable function classes, which is hard to be satisfied in reality. While there are recent works that successfully tackle these strong assumptions, they either require the gap assumptions that only could be satisfied by part of MDPs or use the behavior regularization that makes the optimality of learned policy even intractable. To solve this challenge, we provide finite-sample guarantees for a simple algorithm based on marginalized importance sampling (MIS), showing that sample-efficient offline RL for general MDPs is possible with only a partial coverage dataset and weak realizable function classes given additional side information of a covering distribution. Furthermore, we demonstrate that the covering distribution trades off prior knowledge of the optimal trajectories against the coverage requirement of the dataset, revealing the effect of this inductive bias in the learning processes.

本文研究在函数逼近的情况下，从已有数据集合学习最优策略的离线强化学习问题。研究发现，本文提出的一种简单基于边缘重要采样的算法，可以在数据集合的覆盖率不完整、函数类弱可学习的条件下，通过附加覆盖分布的先验知识来实现理论上的有限次样本保证，同时揭示了学习过程中引入的归纳偏差在覆盖数量与先验知识之间的权衡效应。

增加覆盖分布的离线强化学习