The zeitgeist of the digital era has been dominated by an expanding
integration of Artificial Intelligence~(AI) in a plethora of applications
across various domains. With this expansion, however, questions of the safety
and reliability of these methods come have become more relevant than ever.
Consequently, a run-time ML model safety system has been developed to ensure
the model's operation within the intended context, especially in applications
whose environments are greatly variable such as Autonomous Vehicles~(AVs).
SafeML is a model-agnostic approach for performing such monitoring, using
distance measures based on statistical testing of the training and operational
datasets; comparing them to a predetermined threshold, returning a binary value
whether the model should be trusted in the context of the observed data or be
deemed unreliable. Although a systematic framework exists for this approach,
its performance is hindered by: (1) a dependency on a number of design
parameters that directly affect the selection of a safety threshold and
therefore likely affect its robustness, (2) an inherent assumption of certain
distributions for the training and operational sets, as well as (3) a high
computational complexity for relatively large sets. This work addresses these
limitations by changing the binary decision to a continuous metric.
Furthermore, all data distribution assumptions are made obsolete by
implementing non-parametric approaches, and the computational speed increased
by introducing a new distance measure based on the Empirical Characteristics
Functions~(ECF).

该研究论文提出了 SafeML 这一模型无关的方法，用于监控在自动驾驶等环境中运行的机器学习模型的安全性和可靠性，通过引入基于经验特征函数 (ECF) 的新距离度量方法，消除了二元决策的限制和数据分布假设的依赖，并提高了计算速度。

范围合规性不确定性评估

Scope Compliance Uncertainty Estimate

Reliability estimation of Machine Learning (ML) models is becoming a crucial
subject. This is particularly the case when such \mbox{models} are deployed in
safety-critical applications, as the decisions based on model predictions can
result in hazardous situations. In this regard, recent research has proposed
methods to achieve safe, \mbox{dependable}, and reliable ML systems. One such
method consists of detecting and analyzing distributional shift, and then
measuring how such systems respond to these shifts. This was proposed in
earlier work in SafeML. This work focuses on the use of SafeML for time series
data, and on reliability and robustness estimation of ML-forecasting methods
using statistical distance measures. To this end, distance measures based on
the Empirical Cumulative Distribution Function (ECDF) proposed in SafeML are
explored to measure Statistical-Distance Dissimilarity (SDD) across time
series. We then propose SDD-based Reliability Estimate (StaDRe) and SDD-based
Robustness (StaDRo) measures. With the help of a clustering technique, the
similarity between the statistical properties of data seen during training and
the forecasts is identified. The proposed method is capable of providing a link
between dataset SDD and Key Performance Indicators (KPIs) of the ML models.

本研究探讨了基于 Empirical Cumulative Distribution Function（ECDF）的距离度量方法，以测量时间序列数据中的 ML - 预测方法的可靠性和鲁棒性估计，提出了 SDD-based 可靠性估计（StaDRe）和 SDD-based 鲁棒性估计（StaDRo）措施，并可提供数据集 SDD 与 ML 模型的关键绩效指标之间的联系。