In safe MDP planning, a cost function based on the current state and action
is often used to specify safety aspects. In the real world, often the state
representation used may lack sufficient fidelity to specify such safety
constraints. Operating based on an incomplete model can often produce
unintended negative side effects (NSEs). To address these challenges, first, we
associate safety signals with state-action trajectories (rather than just an
immediate state-action). This makes our safety model highly general. We also
assume categorical safety labels are given for different trajectories, rather
than a numerical cost function, which is harder to specify by the problem
designer. We then employ a supervised learning model to learn such
non-Markovian safety patterns. Second, we develop a Lagrange multiplier method,
which incorporates the safety model and the underlying MDP model in a single
computation graph to facilitate agent learning of safe behaviors. Finally, our
empirical results on a variety of discrete and continuous domains show that
this approach can satisfy complex non-Markovian safety constraints while
optimizing an agent's total returns, is highly scalable, and is also better
than the previous best approach for Markovian NSEs.

本文介绍了一种用于安全 MDP 规划的方法，它基于对状态 - 动作轨迹的安全度量，用监督学习模型学习非马尔可夫安全模式，并通过 Lagrange 乘子方法和计算图优化代理学习安全行为。实验结果表明，该方法可以满足非马尔可夫的安全约束条件，比马尔可夫 NCE 的先前最佳方法更好。