Distribution shift is a major obstacle in offline reinforcement learning,
which necessitates minimizing the discrepancy between the learned policy and
the behavior policy to avoid overestimating rare or unseen actions. Previous
conservative offline RL algorithms struggle to generalize to unseen actions,
despite their success in learning good in-distribution policy. In contrast, we
propose to use the gradient fields of the dataset density generated from a
pre-trained offline RL algorithm to adjust the original actions. We decouple
the conservatism constraints from the policy, thus can benefit wide offline RL
algorithms. As a consequence, we propose the Conservative Denoising Score-based
Algorithm (CDSA) which utilizes the denoising score-based model to model the
gradient of the dataset density, rather than the dataset density itself, and
facilitates a more accurate and efficient method to adjust the action generated
by the pre-trained policy in a deterministic and continuous MDP environment. In
experiments, we show that our approach significantly improves the performance
of baseline algorithms in D4RL datasets, and demonstrate the generalizability
and plug-and-play capability of our model across different pre-trained offline
RL policy in different tasks. We also validate that the agent exhibits greater
risk aversion after employing our method while showcasing its ability to
generalize effectively across diverse tasks.

分布偏移是离线强化学习中的一个主要障碍，为了避免高估罕见或未见动作，需要将学得的策略与行为策略的差异最小化。与以往保守的离线强化学习算法相比，我们提出使用从预训练的离线强化学习算法生成的数据集密度的梯度场来调整原始动作，解耦了保守性约束和策略，从而使广泛的离线强化学习算法受益。我们提出了基于 Conservative Denoising Score 的算法（CDSA），它利用去噪得分模型来建模数据集密度的梯度，而不是数据集密度本身，在确定性和连续的 MDP 环境中提供了更准确和高效的方法来调整预训练策略生成的动作。实验结果表明，我们的方法显著提高了 D4RL 数据集中基线算法的性能，并证明了我们模型在不同任务中基于不同预训练离线强化学习策略的泛化性和即插即用能力，同时验证了代理经过我们方法后表现出更高的风险规避能力，并展示了它在不同任务上的有效泛化能力。