The usage of deep neural networks in safety-critical systems is limited by
our ability to guarantee their correct behavior. Runtime monitors are
components aiming to identify unsafe predictions and discard them before they
can lead to catastrophic consequences. Several recent works on runtime
monitoring have focused on out-of-distribution (OOD) detection, i.e.,
identifying inputs that are different from the training data. In this work, we
argue that OOD detection is not a well-suited framework to design efficient
runtime monitors and that it is more relevant to evaluate monitors based on
their ability to discard incorrect predictions. We call this setting
out-ofmodel-scope detection and discuss the conceptual differences with OOD. We
also conduct extensive experiments on popular datasets from the literature to
show that studying monitors in the OOD setting can be misleading: 1. very good
OOD results can give a false impression of safety, 2. comparison under the OOD
setting does not allow identifying the best monitor to detect errors. Finally,
we also show that removing erroneous training data samples helps to train
better monitors.

本文讨论了深度神经网络在安全临界系统中的应用，提出了基于模型范围的监控方法，并与常用的基于数据范围的监控方法进行了比较，证明了前者具有更高的有效性，并发现删除错误训练数据样本可以帮助更好地训练监控。

拓展域检测不是你所需要的全部

Out-Of-Distribution Detection Is Not All You Need

With the increasing use of Machine Learning (ML) in critical autonomous
systems, runtime monitors have been developed to detect prediction errors and
keep the system in a safe state during operations. Monitors have been proposed
for different applications involving diverse perception tasks and ML models,
and specific evaluation procedures and metrics are used for different contexts.
This paper introduces three unified safety-oriented metrics, representing the
safety benefits of the monitor (Safety Gain), the remaining safety gaps after
using it (Residual Hazard), and its negative impact on the system's performance
(Availability Cost). To compute these metrics, one requires to define two
return functions, representing how a given ML prediction will impact expected
future rewards and hazards. Three use-cases (classification, drone landing, and
autonomous driving) are used to demonstrate how metrics from the literature can
be expressed in terms of the proposed metrics. Experimental results on these
examples show how different evaluation choices impact the perceived performance
of a monitor. As our formalism requires us to formulate explicit safety
assumptions, it allows us to ensure that the evaluation conducted matches the
high-level system requirements.

本文介绍了三种统一的基于安全性的指标，用于衡量机器学习在关键自治系统中应用的安全性，包括安全收益、剩余安全缺陷和对系统性能带来的负面影响，并以分类、无人机着陆和自动驾驶三个用例进行了实验验证。