We develop provably safe and convergent reinforcement learning (RL) algorithms for control of nonlinear dynamical systems, bridging the gap between the hard safety guarantees of control theory and the convergence guarantees of RL theory. Recent advances at the intersection of control and RL follow a two-stage, safety filter approach to enforcing hard safety constraints: model-free RL is used to learn a potentially unsafe controller, whose actions are projected onto safe sets prescribed, for example, by a control barrier function. Though safe, such approaches lose any convergence guarantees enjoyed by the underlying RL methods. In this paper, we develop a single-stage, sampling-based approach to hard constraint satisfaction that learns RL controllers enjoying classical convergence guarantees while satisfying hard safety constraints throughout training and deployment. We validate the efficacy of our approach in simulation, including safe control of a quadcopter in a challenging obstacle avoidance problem, and demonstrate that it outperforms existing benchmarks.

我们开发了一种可证明安全和收敛的非线性动态系统控制的强化学习算法，填补了控制理论的强安全性保证和强化学习理论的收敛保证之间的差距。我们的方法通过单阶段的基于采样的方式，在学习过程和实际应用中满足硬约束条件，同时享有经典的收敛保证，我们在仿真环境中验证了我们方法的有效性，包括一个具有挑战性障碍避免问题的四旋翼无人机的安全控制，并证明其胜过现有的基准模型。

基于采样的安全强化学习在非线性动力系统中的应用