Offline Goal-Conditioned Reinforcement Learning (GCRL) is tasked with
learning to achieve multiple goals in an environment purely from offline
datasets using sparse reward functions. Offline GCRL is pivotal for developing
generalist agents capable of leveraging pre-existing datasets to learn diverse
and reusable skills without hand-engineering reward functions. However,
contemporary approaches to GCRL based on supervised learning and contrastive
learning are often suboptimal in the offline setting. An alternative
perspective on GCRL optimizes for occupancy matching, but necessitates learning
a discriminator, which subsequently serves as a pseudo-reward for downstream
RL. Inaccuracies in the learned discriminator can cascade, negatively
influencing the resulting policy. We present a novel approach to GCRL under a
new lens of mixture-distribution matching, leading to our discriminator-free
method: SMORe. The key insight is combining the occupancy matching perspective
of GCRL with a convex dual formulation to derive a learning objective that can
better leverage suboptimal offline data. SMORe learns scores or unnormalized
densities representing the importance of taking an action at a state for
reaching a particular goal. SMORe is principled and our extensive experiments
on the fully offline GCRL benchmark composed of robot manipulation and
locomotion tasks, including high-dimensional observations, show that SMORe can
outperform state-of-the-art baselines by a significant margin.

基于离线数据集、无初始监督、且没有判别器的分布匹配方法 (SMORe) 在 GCRL 中取得了显著的性能提升。