Proximal policy optimization (PPO) is one of the most popular deep
reinforcement learning (RL) methods, achieving state-of-the-art performance
across a wide range of challenging tasks. However, as a model-free RL method,
the success of PPO relies heavily on the effectiveness of its exploratory
policy search. In this paper, we give an in-depth analysis on the exploration
behavior of PPO, and show that PPO is prone to suffer from the risk of lack of
exploration especially under the case of bad initialization, which may lead to
the failure of training or being trapped in bad local optima. To address these
issues, we proposed a novel policy optimization method, named Trust
Region-Guided PPO (TRGPPO), which adaptively adjusts the clipping range within
the trust region. We formally show that this method not only improves the
exploration ability within the trust region but enjoys a better performance
bound compared to the original PPO as well. Extensive experiments verify the
advantage of the proposed method.

对 Proximal policy optimization 的探索行为进行了深入分析，提出了一种名为 Trust Region-Guided PPO 的新的策略优化方法，通过自适应调整裁剪范围解决了初始条件差的情况下缺乏探索的问题，并证明其相较于原始的 PPO 算法有更好的性能表现。