Machine learning is more and more applied in critical application areas like
health and driver assistance. To minimize the risk of wrong decisions, in such
applications it is necessary to consider the certainty of a classification to
reject uncertain samples. An established tool for this are reject curves that
visualize the trade-off between the number of rejected samples and
classification performance metrics. We argue that common reject curves are too
abstract and hard to interpret by non-experts. We propose Stacked Confusion
Reject Plots (SCORE) that offer a more intuitive understanding of the used data
and the classifier's behavior. We present example plots on artificial Gaussian
data to document the different options of SCORE and provide the code as a
Python package.

对于应用在关键领域（如健康和驾驶辅助）中的机器学习，为了减少错误决策的风险，需要考虑分类的确定性以拒绝不确定样本。我们认为通常的拒绝曲线对非专业人士来说过于抽象和难以理解。因此，我们提出了一种更直观理解数据和分类器行为的 Stacked Confusion Reject Plots (SCORE)。我们在人工高斯数据上展示了示例图，并提供了基于 Python 的代码作为一个软件包。

堆叠混淆拒绝图 (Score)

Stacked Confusion Reject Plots (SCORE)

Given a learning problem with real-world tradeoffs, which cost function
should the model be trained to optimize? This is the metric selection problem
in machine learning. Despite its practical interest, there is limited formal
guidance on how to select metrics for machine learning applications. This
thesis outlines metric elicitation as a principled framework for selecting the
performance metric that best reflects implicit user preferences. Once
specified, the evaluation metric can be used to compare and train models. In
this manuscript, we formalize the problem of Metric Elicitation and devise
novel strategies for eliciting classification performance metrics using
pairwise preference feedback over classifiers. Specifically, we provide novel
strategies for eliciting linear and linear-fractional metrics for binary and
multiclass classification problems, which are then extended to a framework that
elicits group-fair performance metrics in the presence of multiple sensitive
groups. All the elicitation strategies that we discuss are robust to both
finite sample and feedback noise, thus are useful in practice for real-world
applications. Using the tools and the geometric characterizations of the
feasible confusion statistics sets from the binary, multiclass, and
multiclass-multigroup classification setups, we further provide strategies to
elicit from a wider range of complex, modern multiclass metrics defined by
quadratic functions of confusion statistics by exploiting their local linear
structure. From application perspective, we also propose to use the metric
elicitation framework in optimizing complex black box metrics that is amenable
to deep network training. Lastly, to bring theory closer to practice, we
conduct a preliminary real-user study that shows the efficacy of the metric
elicitation framework in recovering the users' preferred performance metric in
a binary classification setup.

本文介绍了 Metric elicitation 作为选择最能反映隐含用户偏好的性能评估指标的原理性框架，并提出使用分类器的成对偏好反馈来激发分类性能指标的新策略，包括用于二元和多类分类问题的线性和线性分式指标的新策略，以及在存在多个敏感组时激发群体公平性能度量的框架的推广，文中也提出了用于优化深度网络训练的黑盒度量的 Metric elicitation 框架，最后还进行了一项初步的真实用户研究，证明了在二值分类设置中 Metric elicitation 框架恢复用户首选性能评估指标的有效性。