Proximal Policy Optimization (PPO) methods learn a policy by iteratively
performing multiple mini-batch optimization epochs of a surrogate objective
with one set of sampled data. Ratio clipping PPO is a popular variant that
clips the probability ratios between the target policy and the policy used to
collect samples. Ratio clipping yields a pessimistic estimate of the original
surrogate objective, and has been shown to be crucial for strong performance.
We show in this paper that such ratio clipping may not be a good option as it
can fail to effectively bound the ratios. Instead, one can directly optimize
the original surrogate objective for multiple epochs; the key is to find a
proper condition to early stop the optimization epoch in each iteration. Our
theoretical analysis sheds light on how to determine when to stop the
optimization epoch, and call the resulting algorithm Early Stopping Policy
Optimization (ESPO). We compare ESPO with PPO across many continuous control
tasks and show that ESPO significantly outperforms PPO. Furthermore, we show
that ESPO can be easily scaled up to distributed training with many workers,
delivering strong performance as well.

本文探讨了比例剪切 PPO 方法的缺陷，提出了一种名为 ESPO 的早停策略优化算法，通过在多个连续控制任务上的比较，发现 ESPO 显著优于 PPO，而且能够轻松扩展到使用多个工作器进行分布式训练。