Biometric Verification (BV) systems often exhibit accuracy disparities across
different demographic groups, leading to biases in BV applications. Assessing
and quantifying these biases is essential for ensuring the fairness of BV
systems. However, existing bias evaluation metrics in BV have limitations, such
as focusing exclusively on match or non-match error rates, overlooking bias on
demographic groups with performance levels falling between the best and worst
performance levels, and neglecting the magnitude of the bias present.
This paper presents an in-depth analysis of the limitations of current bias
evaluation metrics in BV and, through experimental analysis, demonstrates their
contextual suitability, merits, and limitations. Additionally, it introduces a
novel general-purpose bias evaluation measure for BV, the ``Sum of Group Error
Differences (SEDG)''. Our experimental results on controlled synthetic datasets
demonstrate the effectiveness of demographic bias quantification when using
existing metrics and our own proposed measure. We discuss the applicability of
the bias evaluation metrics in a set of simulated demographic bias scenarios
and provide scenario-based metric recommendations. Our code is publicly
available under https://github.com/alaaobeid/SEDG.

生物特征验证（BV）系统往往在不同的人群中显示准确性差异，引发应用中的偏见。评估和量化这些偏见对确保 BV 系统的公正性至关重要。然而，现有的 BV 偏见评估指标存在一些限制，如仅关注匹配或非匹配错误率、忽视性能介于最好和最差性能水平之间人群的偏见以及忽视偏见的强度。本文对 BV 中现有偏见评估指标的限制进行了深入分析，并通过实验分析展示了它们的背景适应性、优点和限制。此外，还引入了一种新颖的通用偏见评估度量指标，即 “组错误差之和（SEDG）”。我们在控制的合成数据集上的实验结果表明，当使用现有指标和我们自己提出的度量时，能够有效量化人口统计偏见。我们讨论了偏见评估指标在一组模拟人口统计偏见场景中的适用性，并提供了基于场景的指标建议。我们的代码在 https://github.com/alaaobeid/SEDG 上公开共享。

组误差和的总和：生物特征验证中偏差评估的关键检查和双度量度

Sum of Group Error Differences: A Critical Examination of Bias  Evaluation in Biometric Verification and a Dual-Metric Measure

Selective classification, in which models can abstain on uncertain
predictions, is a natural approach to improving accuracy in settings where
errors are costly but abstentions are manageable. In this paper, we find that
while selective classification can improve average accuracies, it can
simultaneously magnify existing accuracy disparities between various groups
within a population, especially in the presence of spurious correlations. We
observe this behavior consistently across five vision and NLP datasets.
Surprisingly, increasing abstentions can even decrease accuracies on some
groups. To better understand this phenomenon, we study the margin distribution,
which captures the model's confidences over all predictions. For symmetric
margin distributions, we prove that whether selective classification
monotonically improves or worsens accuracy is fully determined by the accuracy
at full coverage (i.e., without any abstentions) and whether the distribution
satisfies a property we call left-log-concavity. Our analysis also shows that
selective classification tends to magnify full-coverage accuracy disparities.
Motivated by our analysis, we train distributionally-robust models that achieve
similar full-coverage accuracies across groups and show that selective
classification uniformly improves each group on these models. Altogether, our
results suggest that selective classification should be used with care and
underscore the importance of training models to perform equally well across
groups at full coverage.

本文研究了分类模型的选择性分类方法，发现在存在虚假相关性时，选择性分类可以提高平均准确度，但会放大不同群体之间的准确度差异。我们的分析表明，对于对称边缘分布，选择性分类是通过完全覆盖（即没有犹豫）下的准确度和分布是否满足左对数凹性质来确定改善或恶化准确度。基于我们的分析，我们训练了分布稳健模型，并表明选择性分类可以在这些模型上统一改进每个群体的准确性。