Safe reinforcement learning (RL) is still very challenging since it requires
the agent to consider both return maximization and safe exploration. In this
paper, we propose CUP, a Conservative Update Policy algorithm with a
theoretical safety guarantee. We derive the CUP based on the new proposed
performance bounds and surrogate functions. Although using bounds as surrogate
functions to design safe RL algorithms have appeared in some existing works, we
develop them at least three aspects: (i) We provide a rigorous theoretical
analysis to extend the surrogate functions to generalized advantage estimator
(GAE). GAE significantly reduces variance empirically while maintaining a
tolerable level of bias, which is an efficient step for us to design CUP; (ii)
The proposed bounds are tighter than existing works, i.e., using the proposed
bounds as surrogate functions are better local approximations to the objective
and safety constraints. (iii) The proposed CUP provides a non-convex
implementation via first-order optimizers, which does not depend on any convex
approximation. Finally, extensive experiments show the effectiveness of CUP
where the agent satisfies safe constraints. We have opened the source code of
CUP at this https URL.

本文提出了一种基于保守策略更新和理论安全保障的 CUP 算法，并将其与新提出的性能上限和替代函数相结合。在此基础上，文章扩展了替代函数的理论分析，提供了更紧的性能上限，同时基于 GAE 设计了实现更高效的 CUP，最终实验证明了该算法的有效性及其源代码。