Recent advancements in reinforcement learning (RL) have been fueled by large-scale data and deep neural networks, particularly for high-dimensional and complex tasks. Online RL methods like Proximal Policy Optimization (PPO) are effective in dynamic scenarios but require substantial real-time data, posing challenges in resource-constrained or slow simulation environments. Offline RL addresses this by pre-learning policies from large datasets, though its success depends on the quality and diversity of the data. This work proposes a framework that enhances PPO algorithms by incorporating a diffusion model to generate high-quality virtual trajectories for offline datasets. This approach improves exploration and sample efficiency, leading to significant gains in cumulative rewards, convergence speed, and strategy stability in complex tasks. Our contributions are threefold: we explore the potential of diffusion models in RL, particularly for offline datasets, extend the application of online RL to offline environments, and experimentally validate the performance improvements of PPO with diffusion models. These findings provide new insights and methods for applying RL to high-dimensional, complex tasks. Finally, we open-source our code at https://github.com/TianciGao/DiffPPO

本研究针对在线强化学习方法在资源受限或模拟环境中对实时数据的高需求这一问题，提出了一个框架，将扩散模型与近端策略优化算法结合，以生成高质量的虚拟轨迹，从而增强样本效率和探索性。实验结果表明，该方法在复杂任务中显著提升了累积奖励、收敛速度和策略稳定性，为高维复杂任务中的强化学习提供了新的理论与实践方法。

通过集成扩散模型和近端策略优化提高强化学习的样本效率和探索性