Any autonomous controller will be unsafe in some situations. The ability to
quantitatively identify when these unsafe situations are about to occur is
crucial for drawing timely human oversight in, e.g., freight transportation
applications. In this work, we demonstrate that the true criticality of an
agent's situation can be robustly defined as the mean reduction in reward given
some number of random actions. Proxy criticality metrics that are computable in
real-time (i.e., without actually simulating the effects of random actions) can
be compared to the true criticality, and we show how to leverage these proxy
metrics to generate safety margins, which directly tie the consequences of
potentially incorrect actions to an anticipated loss in overall performance. We
evaluate our approach on learned policies from APE-X and A3C within an Atari
environment, and demonstrate how safety margins decrease as agents approach
failure states. The integration of safety margins into programs for monitoring
deployed agents allows for the real-time identification of potentially
catastrophic situations.

任何自主控制器在某些情况下都可能不安全，定量识别这些不安全情况的能力对于及时引起人类监督至关重要。本研究展示了真正危急情况的临界性可以被鲁棒地定义为随机行动导致的奖励减少的平均值。可以将实时计算的代理临界性度量与真正的临界性进行比较，并展示如何利用这些代理度量生成安全裕度，直接将潜在错误行动的后果与整体性能损失联系起来。我们在 Atari 环境中评估了来自 APE-X 和 A3C 的学习策略，并展示了安全裕度随着代理逼近失败状态而减少。将安全裕度整合到监测已部署代理的程序中，可以实时识别潜在的灾难性情况。