Offline reinforcement learning (RL) refers to the problem of learning
policies from a static dataset of environment interactions. Offline RL enables
extensive use and re-use of historical datasets, while also alleviating safety
concerns associated with online exploration, thereby expanding the real-world
applicability of RL. Most prior work in offline RL has focused on tasks with
compact state representations. However, the ability to learn directly from rich
observation spaces like images is critical for real-world applications such as
robotics. In this work, we build on recent advances in model-based algorithms
for offline RL, and extend them to high-dimensional visual observation spaces.
Model-based offline RL algorithms have achieved state of the art results in
state based tasks and have strong theoretical guarantees. However, they rely
crucially on the ability to quantify uncertainty in the model predictions,
which is particularly challenging with image observations. To overcome this
challenge, we propose to learn a latent-state dynamics model, and represent the
uncertainty in the latent space. Our approach is both tractable in practice and
corresponds to maximizing a lower bound of the ELBO in the unknown POMDP. In
experiments on a range of challenging image-based locomotion and manipulation
tasks, we find that our algorithm significantly outperforms previous offline
model-free RL methods as well as state-of-the-art online visual model-based RL
methods. Moreover, we also find that our approach excels on an image-based
drawer closing task on a real robot using a pre-existing dataset. All results
including videos can be found online at this https URL .

该研究提出了一种基于模型的离线 RL 算法，该算法可扩展应用于高维视觉观测空间，通过学习潜在状态动力学模型并在潜在空间中表示不确定性来克服图像观测的挑战，并在多项具有挑战性的基于图像的移动和操纵任务中取得了优异的表现。