Safe exploration is essential for the practical use of reinforcement learning (RL) in many real-world scenarios. In this paper, we present a generalized safe exploration (GSE) problem as a unified formulation of common safe exploration problems. We then propose a solution of the GSE problem in the form of a meta-algorithm for safe exploration, MASE, which combines an unconstrained RL algorithm with an uncertainty quantifier to guarantee safety in the current episode while properly penalizing unsafe explorations before actual safety violation to discourage them in future episodes. The advantage of MASE is that we can optimize a policy while guaranteeing with a high probability that no safety constraint will be violated under proper assumptions. Specifically, we present two variants of MASE with different constructions of the uncertainty quantifier: one based on generalized linear models with theoretical guarantees of safety and near-optimality, and another that combines a Gaussian process to ensure safety with a deep RL algorithm to maximize the reward. Finally, we demonstrate that our proposed algorithm achieves better performance than state-of-the-art algorithms on grid-world and Safety Gym benchmarks without violating any safety constraints, even during training.

在本篇论文中，我们提出了一个名为广义安全探索（GSE）的问题，并使用元算法MASE解决了这个问题，该算法结合了无限制的强化学习算法和不确定性量化器，以确保当前回合的安全性，同时在实际安全违规之前对不安全的探索进行适当的惩罚，以防止它们在未来回合中发生。MASE算法的优势在于我们可以在合理假设下优化策略，同时以很高的概率保证不违反任何安全约束。最后，我们演示了我们提出的算法在方格世界和Safety Gym基准测试上取得了比现有算法更好的性能，且即使在训练过程中也没有违反任何安全约束。

强化学习中的安全探索：广义形式与算法