Offline reinforcement learning (RL) aims to learn policies from static
datasets of previously collected trajectories. Existing methods for offline RL
either constrain the learned policy to the support of offline data or utilize
model-based virtual environments to generate simulated rollouts. However, these
methods suffer from (i) poor generalization to unseen states; and (ii) trivial
improvement from low-qualified rollout simulation. In this paper, we propose
offline trajectory generalization through world transformers for offline
reinforcement learning (OTTO). Specifically, we use casual Transformers, a.k.a.
World Transformers, to predict state dynamics and the immediate reward. Then we
propose four strategies to use World Transformers to generate high-rewarded
trajectory simulation by perturbing the offline data. Finally, we jointly use
offline data with simulated data to train an offline RL algorithm. OTTO serves
as a plug-in module and can be integrated with existing offline RL methods to
enhance them with better generalization capability of transformers and
high-rewarded data augmentation. Conducting extensive experiments on D4RL
benchmark datasets, we verify that OTTO significantly outperforms
state-of-the-art offline RL methods.

通过世界转换器进行线下强化学习的离线轨迹泛化方法（OTTO）在 D4RL 基准数据集上验证了其相对于最先进的线下强化学习方法具有显著优势。