Selective Classification, wherein models can reject low-confidence
predictions, promises reliable translation of machine-learning based
classification systems to real-world scenarios such as clinical diagnostics.
While current evaluation of these systems typically assumes fixed working
points based on pre-defined rejection thresholds, methodological progress
requires benchmarking the general performance of systems akin to the
$\mathrm{AUROC}$ in standard classification. In this work, we define 5
requirements for multi-threshold metrics in selective classification regarding
task alignment, interpretability, and flexibility, and show how current
approaches fail to meet them. We propose the Area under the Generalized Risk
Coverage curve ($\mathrm{AUGRC}$), which meets all requirements and can be
directly interpreted as the average risk of undetected failures. We empirically
demonstrate the relevance of $\mathrm{AUGRC}$ on a comprehensive benchmark
spanning 6 data sets and 13 confidence scoring functions. We find that the
proposed metric substantially changes metric rankings on 5 out of the 6 data
sets.

选择性分类方法可以在低置信度预测时拒绝，可靠地将基于机器学习的分类系统应用于临床诊断等实际场景。本研究定义了多阈值评估度量在选择性分类中的 5 个要求，涉及任务对齐、可解释性和灵活性，并展示了当前方法未能满足这些要求。我们提出了广义风险覆盖曲线下的面积（AUGRC），该曲线满足所有要求，并可直接解释为未检测到故障的平均风险。通过综合评估 6 个数据集和 13 个置信度评分函数的广泛基准，我们在实证上证明了 AUGRC 的相关性。我们发现，在这 6 个数据集中，所提出的度量方法显著改变了度量排名中的 5 个数据集。