Safe exploration presents a major challenge in reinforcement learning (RL):
when active data collection requires deploying partially trained policies, we
must ensure that these policies avoid catastrophically unsafe regions, while
still enabling trial and error learning. In this paper, we target the problem
of safe exploration in RL by learning a conservative safety estimate of
environment states through a critic, and provably upper bound the likelihood of
catastrophic failures at every training iteration. We theoretically
characterize the tradeoff between safety and policy improvement, show that the
safety constraints are likely to be satisfied with high probability during
training, derive provable convergence guarantees for our approach, which is no
worse asymptotically than standard RL, and demonstrate the efficacy of the
proposed approach on a suite of challenging navigation, manipulation, and
locomotion tasks. Empirically, we show that the proposed approach can achieve
competitive task performance while incurring significantly lower catastrophic
failure rates during training than prior methods. Videos are at this url
this https URL

本文介绍了一种基于保守安全估计的强化学习安全探索方法，通过批判学习环境状态的保守安全估计，从理论上上界限制了灾难性失败概率，实验证明该方法在解决导航、操作和运动任务时达到了具有竞争力的任务性能，同步显著降低了灾难性失败率。