Proximal policy optimization (PPO) is one of the most successful deep
reinforcement-learning methods, achieving state-of-the-art performance across a
wide range of challenging tasks. However, its optimization behavior is still
far from being fully understood. In this paper, we show that PPO could neither
strictly restrict the likelihood ratio as it attempts to do nor enforce a
well-defined trust region constraint, which means that it may still suffer from
the risk of performance instability. To address this issue, we present an
enhanced PPO method, named Truly PPO. Two critical improvements are made in our
method: 1) it adopts a new clipping function to support a rollback behavior to
restrict the difference between the new policy and the old one; 2) the
triggering condition for clipping is replaced with a trust region-based one,
such that optimizing the resulted surrogate objective function provides
guaranteed monotonic improvement of the ultimate policy performance. It seems,
by adhering more truly to making the algorithm proximal - confining the policy
within the trust region, the new algorithm improves the original PPO on both
sample efficiency and performance.

本文介绍了一种名为 Truly PPO 的增强 PPO 方法，针对 PPO 在优化行为方面存在的问题进行了改进，通过使用新的剪辑函数来支持回滚行为，使用基于可信区域的触发条件替换剪辑的触发条件，从而提供了保证的拟态策略性能单调改进，从而改善了 PPO 在样本效率和性能方面的表现。