We observe that several existing policy gradient methods (such as vanilla
policy gradient, PPO, A2C) may suffer from overly large gradients when the
current policy is close to deterministic (even in some very simple
environments), leading to an unstable training process. To address this issue,
we propose a new method, called \emph{target distribution learning} (TDL), for
policy improvement in reinforcement learning. TDL alternates between proposing
a target distribution and training the policy network to approach the target
distribution. TDL is more effective in constraining the KL divergence between
updated policies, and hence leads to more stable policy improvements over
iterations. Our experiments show that TDL algorithms perform comparably to (or
better than) state-of-the-art algorithms for most continuous control tasks in
the MuJoCo environment while being more stable in training.

本研究提出了新的强化学习方法 —— 目标分布学习 (TDL)，通过交替提出目标分布和训练策略网络来解决现有策略梯度方法可能出现过大梯度和训练不稳定的问题，实验发现这种方法能得到类似或更好的结果，并且更加稳定。