Due to its training stability and strong expression, the diffusion model has
attracted considerable attention in offline reinforcement learning. However,
several challenges have also come with it: 1) The demand for a large number of
diffusion steps makes the diffusion-model-based methods time inefficient and
limits their applications in real-time control; 2) How to achieve policy
improvement with accurate guidance for diffusion model-based policy is still an
open problem. Inspired by the consistency model, we propose a novel
time-efficiency method named Consistency Policy with Q-Learning (CPQL), which
derives action from noise by a single step. By establishing a mapping from the
reverse diffusion trajectories to the desired policy, we simultaneously address
the issues of time efficiency and inaccurate guidance when updating diffusion
model-based policy with the learned Q-function. We demonstrate that CPQL can
achieve policy improvement with accurate guidance for offline reinforcement
learning, and can be seamlessly extended for online RL tasks. Experimental
results indicate that CPQL achieves new state-of-the-art performance on 11
offline and 21 online tasks, significantly improving inference speed by nearly
45 times compared to Diffusion-QL. We will release our code later.

通过一步将噪声转化为动作，我们提出了一种名为 CPQL 的新型时间效率方法，解决了扩散模型在更新时的时间效率和准确性指导方面的问题，从而实现了脱机强化学习的策略改进，并可以无缝地扩展到在线强化学习任务中，最终实验结果表明，CPQL 在 11 个脱机任务和 21 个在线任务中取得了新的最高性能，推理速度相比 Diffusion-QL 提高了近 45 倍。