Partially observable Markov decision processes (POMDPs) have been widely used
in many robotic applications for sequential decision-making under uncertainty.
POMDP online planning algorithms such as Partially Observable Monte-Carlo
Planning (POMCP) can solve very large POMDPs with the goal of maximizing the
expected return. But the resulting policies cannot provide safety guarantees
that are imperative for real-world safety-critical tasks (e.g., autonomous
driving). In this work, we consider safety requirements represented as
almost-sure reach-avoid specifications (i.e., the probability to reach a set of
goal states is one and the probability to reach a set of unsafe states is
zero). We compute shields that restrict unsafe actions violating almost-sure
reach-avoid specifications. We then integrate these shields into the POMCP
algorithm for safe POMDP online planning. We propose four distinct shielding
methods, differing in how the shields are computed and integrated, including
factored variants designed to improve scalability. Experimental results on a
set of benchmark domains demonstrate that the proposed shielding methods
successfully guarantee safety (unlike the baseline POMCP without shielding) on
large POMDPs, with negligible impact on the runtime for online planning.

本文研究如何通过引入安全限制来解决部分可观察的马尔可夫决策过程（POMDPs）中的不确定性问题，其中通过计算和整合防护盾到在线规划算法（POMCP）中以确保安全性。实验结果表明，该方法在大型 POMDPs 中成功地保证了安全性，并对在线规划的运行时间几乎没有影响。

安全的 POMDP 在线规划方法之护盾技术

Safe POMDP Online Planning via Shielding

Safe exploration is a common problem in reinforcement learning (RL) that aims
to prevent agents from making disastrous decisions while exploring their
environment. A family of approaches to this problem assume domain knowledge in
the form of a (partial) model of this environment to decide upon the safety of
an action. A so-called shield forces the RL agent to select only safe actions.
However, for adoption in various applications, one must look beyond enforcing
safety and also ensure the applicability of RL with good performance. We extend
the applicability of shields via tight integration with state-of-the-art deep
RL, and provide an extensive, empirical study in challenging, sparse-reward
environments under partial observability. We show that a carefully integrated
shield ensures safety and can improve the convergence rate and final
performance of RL agents. We furthermore show that a shield can be used to
bootstrap state-of-the-art RL agents: they remain safe after initial learning
in a shielded setting, allowing us to disable a potentially too conservative
shield eventually.

研究了强化学习中的安全探索问题，提出了一种叫做 shield 的方法来确保代理人只进行安全的操作，并深度结合使用 state-of-the-art deep RL 来提高 RL 代理人的性能，实验证明这种方法可以提高收敛速度和最终表现，并可用于引导 state-of-the-art RL 代理人。