Existing offline reinforcement learning (RL) methods face a few major
challenges, particularly the distributional shift between the learned policy
and the behavior policy. Offline Meta-RL is emerging as a promising approach to
address these challenges, aiming to learn an informative meta-policy from a
collection of tasks. Nevertheless, as shown in our empirical studies, offline
Meta-RL could be outperformed by offline single-task RL methods on tasks with
good quality of datasets, indicating that a right balance has to be delicately
calibrated between "exploring" the out-of-distribution state-actions by
following the meta-policy and "exploiting" the offline dataset by staying close
to the behavior policy. Motivated by such empirical analysis, we explore
model-based offline Meta-RL with regularized Policy Optimization (MerPO), which
learns a meta-model for efficient task structure inference and an informative
meta-policy for safe exploration of out-of-distribution state-actions. In
particular, we devise a new meta-Regularized model-based Actor-Critic (RAC)
method for within-task policy optimization, as a key building block of MerPO,
using conservative policy evaluation and regularized policy improvement; and
the intrinsic tradeoff therein is achieved via striking the right balance
between two regularizers, one based on the behavior policy and the other on the
meta-policy. We theoretically show that the learnt policy offers guaranteed
improvement over both the behavior policy and the meta-policy, thus ensuring
the performance improvement on new tasks via offline Meta-RL. Experiments
corroborate the superior performance of MerPO over existing offline Meta-RL
methods.

该研究论文介绍了一种基于模型的元强化学习方法 ——MerPO，使用正则化策略优化来实现任务结构推断和元策略安全探索。该方法通过探究 “探索” 元策略的分布情况和 “利用” 离线数据集的紧密度之间的平衡，对元强化学习算法进行了改进，并在实验中取得了优异的表现。