Failure detection (FD) in AI systems is a crucial safeguard for the
deployment for safety-critical tasks. The common evaluation method of FD
performance is the Risk-coverage (RC) curve, which reveals the trade-off
between the data coverage rate and the performance on accepted data. One common
way to quantify the RC curve by calculating the area under the RC curve.
However, this metric does not inform on how suited any method is for FD, or
what the optimal coverage rate should be. As FD aims to achieve higher
performance with fewer data discarded, evaluating with partial coverage
excluding the most uncertain samples is more intuitive and meaningful than full
coverage. In addition, there is an optimal point in the coverage where the
model could achieve ideal performance theoretically. We propose the Excess Area
Under the Optimal RC Curve (E-AUoptRC), with the area in coverage from the
optimal point to the full coverage. Further, the model performance at this
optimal point can represent both model learning ability and calibration. We
propose it as the Trust Index (TI), a complementary evaluation metric to the
overall model accuracy. We report extensive experiments on three benchmark
image datasets with ten variants of transformer and CNN models. Our results
show that our proposed methods can better reflect the model trustworthiness
than existing evaluation metrics. We further observe that the model with high
overall accuracy does not always yield the high TI, which indicates the
necessity of the proposed Trust Index as a complementary metric to the model
overall accuracy. The code are available at
https://github.com/AoShuang92/optimal_risk.

AI 系统中的失败检测是安全关键任务部署的重要保障，通过计算最优风险覆盖曲线下的区域和模型在此最优点上的性能，我们提出了衡量失败检测方法适用性和模型可信度的指标，通过实验验证了其有效性。

度量模型可信度的实证最优风险用于故障检测

Empirical Optimal Risk to Quantify Model Trustworthiness for Failure  Detection

Data collected from the real world tends to be biased, unbalanced, and at
risk of exposing sensitive and private information. This reality has given rise
to the idea of creating synthetic datasets to alleviate risk, bias, harm, and
privacy concerns inherent in the real data. This concept relies on Generative
AI models to produce unbiased, privacy-preserving synthetic data while being
true to the real data. In this new paradigm, how can we tell if this approach
delivers on its promises? We present an auditing framework that offers a
holistic assessment of synthetic datasets and AI models trained on them,
centered around bias and discrimination prevention, fidelity to the real data,
utility, robustness, and privacy preservation. We showcase our framework by
auditing multiple generative models on diverse use cases, including education,
healthcare, banking, human resources, and across different modalities, from
tabular, to time-series, to natural language. Our use cases demonstrate the
importance of a holistic assessment in order to ensure compliance with
socio-technical safeguards that regulators and policymakers are increasingly
enforcing. For this purpose, we introduce the trust index that ranks multiple
synthetic datasets based on their prescribed safeguards and their desired
trade-offs. Moreover, we devise a trust-index-driven model selection and
cross-validation procedure via auditing in the training loop that we showcase
on a class of transformer models that we dub TrustFormers, across different
modalities. This trust-driven model selection allows for controllable trust
trade-offs in the resulting synthetic data. We instrument our auditing
framework with workflows that connect different stakeholders from model
development to audit and certification via a synthetic data auditing report.

本文提出一个审核框架，围绕偏见和歧视预防、与真实数据的一致性、效用、鲁棒性和隐私保护进行综合评估，并引入信任指数对多个合成数据集进行排名，通过审核和认证报告连接不同利益相关者。