Due to its training stability and strong expression, the diffusion model has attracted considerable attention in offline reinforcement learning. However, several challenges have also come with it: 1) The demand for a large number of diffusion steps makes the diffusion-model-based methods time inefficient and limits their applications in real-time control; 2) How to achieve policy improvement with accurate guidance for diffusion model-based policy is still an open problem. Inspired by the consistency model, we propose a novel time-efficiency method named Consistency Policy with Q-Learning (CPQL), which derives action from noise by a single step. By establishing a mapping from the reverse diffusion trajectories to the desired policy, we simultaneously address the issues of time efficiency and inaccurate guidance when updating diffusion model-based policy with the learned Q-function. We demonstrate that CPQL can achieve policy improvement with accurate guidance for offline reinforcement learning, and can be seamlessly extended for online RL tasks. Experimental results indicate that CPQL achieves new state-of-the-art performance on 11 offline and 21 online tasks, significantly improving inference speed by nearly 45 times compared to Diffusion-QL. We will release our code later.

通过一步将噪声转化为动作，我们提出了一种名为CPQL的新型时间效率方法，解决了扩散模型在更新时的时间效率和准确性指导方面的问题，从而实现了脱机强化学习的策略改进，并可以无缝地扩展到在线强化学习任务中，最终实验结果表明，CPQL在11个脱机任务和21个在线任务中取得了新的最高性能，推理速度相比Diffusion-QL提高了近45倍。

使用一致性策略提升连续控制