Model-based reinforcement learning (RL), which learns environment model from
offline dataset and generates more out-of-distribution model data, has become
an effective approach to the problem of distribution shift in offline RL. Due
to the gap between the learned and actual environment, conservatism should be
incorporated into the algorithm to balance accurate offline data and imprecise
model data. The conservatism of current algorithms mostly relies on model
uncertainty estimation. However, uncertainty estimation is unreliable and leads
to poor performance in certain scenarios, and the previous methods ignore
differences between the model data, which brings great conservatism. Therefore,
this paper proposes a milDly cOnservative Model-bAsed offlINe RL algorithm
(DOMAIN) without estimating model uncertainty to address the above issues.
DOMAIN introduces adaptive sampling distribution of model samples, which can
adaptively adjust the model data penalty. In this paper, we theoretically
demonstrate that the Q value learned by the DOMAIN outside the region is a
lower bound of the true Q value, the DOMAIN is less conservative than previous
model-based offline RL algorithms and has the guarantee of security policy
improvement. The results of extensive experiments show that DOMAIN outperforms
prior RL algorithms on the D4RL dataset benchmark, and achieves better
performance than other RL algorithms on tasks that require generalization.

这篇论文提出了一种不需要模型不确定性估计的温和保守型基于模型的离线强化学习算法 (DOMAIN)，通过引入模型样本的自适应抽样分布来调整模型数据惩罚，理论上证明了该算法在区域外学习到的 Q 值是真实 Q 值的下界，与先前的基于模型的离线强化学习算法相比，DOMAIN 是较保守的，并且具有安全策略改进的保证。大量实验证明 DOMAIN 在 D4RL 数据集基准上优于先前的强化学习算法，并且在需要泛化的任务上比其他强化学习算法表现更好。