In offline RL, constraining the learned policy to remain close to the data is
essential to prevent the policy from outputting out-of-distribution (OOD)
actions with erroneously overestimated values. In principle, generative
adversarial networks (GAN) can provide an elegant solution to do so, with the
discriminator directly providing a probability that quantifies distributional
shift. However, in practice, GAN-based offline RL methods have not performed as
well as alternative approaches, perhaps because the generator is trained to
both fool the discriminator and maximize return -- two objectives that can be
at odds with each other. In this paper, we show that the issue of conflicting
objectives can be resolved by training two generators: one that maximizes
return, with the other capturing the ``remainder'' of the data distribution in
the offline dataset, such that the mixture of the two is close to the behavior
policy. We show that not only does having two generators enable an effective
GAN-based offline RL method, but also approximates a support constraint, where
the policy does not need to match the entire data distribution, but only the
slice of the data that leads to high long term performance. We name our method
DASCO, for Dual-Generator Adversarial Support Constrained Offline RL. On
benchmark tasks that require learning from sub-optimal data, DASCO
significantly outperforms prior methods that enforce distribution constraint.

本文介绍了一种名为 DASCO 的新方法，用于解决离线强化学习 (GAN) 中存在的矛盾目标问题，该方法采用了双生成器，其中一个最大化回报，另一个捕捉不属于行为策略的数据分布的 “剩余部分”，这两者的混合物接近于行为策略，并近似支持约束，通过在子优数据上进行基准测试，表明 DASCO 明显优于之前的方法，实现了分布约束。