Ensuring safety is a crucial challenge when deploying reinforcement learning (RL) to real-world systems. We develop confidence-based safety filters, a control-theoretic approach for certifying state safety constraints for nominal policies learned via standard RL techniques, based on probabilistic dynamics models. Our approach is based on a reformulation of state constraints in terms of cost functions, reducing safety verification to a standard RL task. By exploiting the concept of hallucinating inputs, we extend this formulation to determine a "backup" policy that is safe for the unknown system with high probability. Finally, the nominal policy is minimally adjusted at every time step during a roll-out towards the backup policy, such that safe recovery can be guaranteed afterwards. We provide formal safety guarantees, and empirically demonstrate the effectiveness of our approach.

在强化学习应用于现实系统时，确保安全是一个关键的挑战。因此，我们通过概率动力学模型提供一种基于控制理论的置信度安全过滤器方法，用于认证通过标准强化学习技术学习的名义策略的状态安全约束条件，将安全验证降低到标准强化学习任务。利用幻想输入的概念，我们将这种方法扩展到确定对未知系统具有高概率安全的“备份”策略。最后，在朝向备份策略的滚动过程中，每个时间步骤最小调整名义策略，以保证安全恢复。我们提供了正式的安全保证，并在实验中证明了我们方法的有效性。

基于置信度过滤器的安全强化学习