Ensuring safety is important for the practical deployment of reinforcement learning (RL). Various challenges must be addressed, such as handling stochasticity in the environments, providing rigorous guarantees of persistent state-wise safety satisfaction, and avoiding overly conservative behaviors that sacrifice performance. We propose a new framework, Reachability Estimation for Safe Policy Optimization (RESPO), for safety-constrained RL in general stochastic settings. In the feasible set where there exist violation-free policies, we optimize for rewards while maintaining persistent safety. Outside this feasible set, our optimization produces the safest behavior by guaranteeing entrance into the feasible set whenever possible with the least cumulative discounted violations. We introduce a class of algorithms using our novel reachability estimation function to optimize in our proposed framework and in similar frameworks such as those concurrently handling multiple hard and soft constraints. We theoretically establish that our algorithms almost surely converge to locally optimal policies of our safe optimization framework. We evaluate the proposed methods on a diverse suite of safe RL environments from Safety Gym, PyBullet, and MuJoCo, and show the benefits in improving both reward performance and safety compared with state-of-the-art baselines.

确保安全对于强化学习的实际部署非常重要。我们提出了一个新的框架，用于在一般的随机环境中进行安全约束的强化学习。我们的算法在最优化奖励的同时保持持续的安全特性，通过最小化累积折扣违规行为来产生最安全的行为，并基于可达性估计来优化我们提出的框架。我们在多个安全强化学习环境上评估了该方法，并与当前最先进的基准方法进行了比较，结果显示了在提高奖励性能和安全性方面的优势。

安全强化学习中的迭代可达性估计