Offline meta-reinforcement learning (meta-RL) methods, which adapt to unseen
target tasks with prior experience, are essential in robot control tasks.
Current methods typically utilize task contexts and skills as prior experience,
where task contexts are related to the information within each task and skills
represent a set of temporally extended actions for solving subtasks. However,
these methods still suffer from limited performance when adapting to unseen
target tasks, mainly because the learned prior experience lacks generalization,
i.e., they are unable to extract effective prior experience from meta-training
tasks by exploration and learning of continuous latent spaces. We propose a
framework called decoupled meta-reinforcement learning (DCMRL), which (1)
contrastively restricts the learning of task contexts through pulling in
similar task contexts within the same task and pushing away different task
contexts of different tasks, and (2) utilizes a Gaussian quantization
variational autoencoder (GQ-VAE) for clustering the Gaussian distributions of
the task contexts and skills respectively, and decoupling the exploration and
learning processes of their spaces. These cluster centers which serve as
representative and discrete distributions of task context and skill are stored
in task context codebook and skill codebook, respectively. DCMRL can acquire
generalizable prior experience and achieve effective adaptation to unseen
target tasks during the meta-testing phase. Experiments in the navigation and
robot manipulation continuous control tasks show that DCMRL is more effective
than previous meta-RL methods with more generalizable prior experience.

离线元强化学习方法（即通过先前经验适应未见的目标任务）在机器人控制任务中至关重要。本文提出了一种名为分离式元强化学习（DCMRL）的框架，通过对任务上下文和技能的学习和探索进行量化和离散化，从而获取可推广的先前经验并在元测试阶段实现对未见目标任务的有效适应。实验证明，DCMRL 比先前的元强化学习方法具有更具推广性的先前经验，并在导航和机器人操纵连续控制任务中更加有效。