Offline reinforcement learning (RL) leverages pre-collected datasets to train
optimal policies. Diffusion Q-Learning (DQL), introducing diffusion models as a
powerful and expressive policy class, significantly boosts the performance of
offline RL. However, its reliance on iterative denoising sampling to generate
actions slows down both training and inference. While several recent attempts
have tried to accelerate diffusion-QL, the improvement in training and/or
inference speed often results in degraded performance. In this paper, we
introduce a dual policy approach, Diffusion Trusted Q-Learning (DTQL), which
comprises a diffusion policy for pure behavior cloning and a practical one-step
policy. We bridge the two polices by a newly introduced diffusion trust region
loss. The diffusion policy maintains expressiveness, while the trust region
loss directs the one-step policy to explore freely and seek modes within the
region defined by the diffusion policy. DTQL eliminates the need for iterative
denoising sampling during both training and inference, making it remarkably
computationally efficient. We evaluate its effectiveness and algorithmic
characteristics against popular Kullback-Leibler (KL) based distillation
methods in 2D bandit scenarios and gym tasks. We then show that DTQL could not
only outperform other methods on the majority of the D4RL benchmark tasks but
also demonstrate efficiency in training and inference speeds. The PyTorch
implementation will be made available.

离线强化学习中的扩散信任 Q 学习方法（DTQL）通过引入扩散模型作为一个强大和有表达力的策略类，消除了训练和推理过程中迭代去噪采样的需要，大大提高了计算效率，并在多个基准任务中展现了优越的性能和算法特性。