We propose Adversarially Trained Actor Critic (ATAC), a new model-free
algorithm for offline reinforcement learning (RL) under insufficient data
coverage, based on the concept of relative pessimism. ATAC is designed as a
two-player Stackelberg game: A policy actor competes against an adversarially
trained value critic, who finds data-consistent scenarios where the actor is
inferior to the data-collection behavior policy. We prove that, when the actor
attains no regret in the two-player game, running ATAC produces a policy that
provably 1) outperforms the behavior policy over a wide range of
hyperparameters that control the degree of pessimism, and 2) competes with the
best policy covered by data with appropriately chosen hyperparameters. Compared
with existing works, notably our framework offers both theoretical guarantees
for general function approximation and a deep RL implementation scalable to
complex environments and large datasets. In the D4RL benchmark, ATAC
consistently outperforms state-of-the-art offline RL algorithms on a range of
continuous control tasks.

本文提出 Adversarially Trained Actor Critic (ATAC) 算法解决数据不足下的离线强化学习问题，通过相对悲观的方式，在两个玩家之间进行 Stackelberg 博弈，找到 “差策略” 并优化策略，该算法在复杂环境和大数据集上的实现得到理论上的保障，并在 D4RL 基准测试中表现优秀。