In this work, we study offline reinforcement learning (RL) with zero-shot generalization property (ZSG), where the agent has access to an offline dataset including experiences from different environments, and the goal of the agent is to train a policy over the training environments which performs well on test environments without further interaction. Existing work showed that classical offline RL fails to generalize to new, unseen environments. We propose pessimistic empirical risk minimization (PERM) and pessimistic proximal policy optimization (PPPO), which leverage pessimistic policy evaluation to guide policy learning and enhance generalization. We show that both PERM and PPPO are capable of finding a near-optimal policy with ZSG. Our result serves as a first step in understanding the foundation of the generalization phenomenon in offline reinforcement learning.

本研究解决了离线强化学习（RL）中缺乏零样本泛化能力的问题。提出了悲观经验风险最小化（PERM）和悲观近端策略优化（PPPO）两种新方法，通过悲观策略评估引导策略学习，从而提升泛化能力。研究表明，PERM和PPPO能够找到近似最优的策略，标志着对离线强化学习中泛化现象基础的初步理解。

离线强化学习中的可证明零样本泛化