Offline reinforcement learning (RL) extends the paradigm of classical RL algorithms to purely learning from static datasets, without interacting with the underlying environment during the learning process. A key challenge of offline RL is the instability of policy training, caused by the mismatch between the distribution of the offline data and the undiscounted stationary state-action distribution of the learned policy. To avoid the detrimental impact of distribution mismatch, we regularize the undiscounted stationary distribution of the current policy towards the offline data during the policy optimization process. Further, we train a dynamics model to both implement this regularization and better estimate the stationary distribution of the current policy, reducing the error induced by distribution mismatch. On a wide range of continuous-control offline RL datasets, our method indicates competitive performance, which validates our algorithm. The code is publicly available.

该研究针对离线强化学习中策略训练不稳定的问题，通过对当前策略的无折扣平稳分布进行正则化，并训练动力学模型以实现该正则化和更好地估计当前策略的平稳分布，从而减少分布不匹配引起的误差，并在广泛的连续控制离线RL数据集上展现出竞争性的性能。

将基于模型策略的稳态分布规范化以稳定离线强化学习