Machine learning (ML) models used in prediction and classification tasks may
display performance disparities across population groups determined by
sensitive attributes (e.g., race, sex, age). We consider the problem of
evaluating the performance of a fixed ML model across population groups defined
by multiple sensitive attributes (e.g., race and sex and age). Here, the sample
complexity for estimating the worst-case performance gap across groups (e.g.,
the largest difference in error rates) increases exponentially with the number
of group-denoting sensitive attributes. To address this issue, we propose an
approach to test for performance disparities based on Conditional Value-at-Risk
(CVaR). By allowing a small probabilistic slack on the groups over which a
model has approximately equal performance, we show that the sample complexity
required for discovering performance violations is reduced exponentially to be
at most upper bounded by the square root of the number of groups. As a
byproduct of our analysis, when the groups are weighted by a specific prior
distribution, we show that R\'enyi entropy of order $2/3$ of the prior
distribution captures the sample complexity of the proposed CVaR test
algorithm. Finally, we also show that there exists a non-i.i.d. data collection
strategy that results in a sample complexity independent of the number of
groups.

通过允许模型在性能近乎相等的群体上产生微小概率差距的方法，基于条件风险价值 (CVaR) 提出了一种检测性能差异的方法；同时证明了具有特定先验分布权重的情况下，Rényi 熵阶数为 2/3 的先验分布可以捕获所提出的 CVaR 测试算法的样本复杂性。

通过条件风险值测试进行多组公平性评估

Multi-Group Fairness Evaluation via Conditional Value-at-Risk Testing

Understanding the cumulative effect of multiple fairness enhancing
interventions at different stages of the machine learning (ML) pipeline is a
critical and underexplored facet of the fairness literature. Such knowledge can
be valuable to data scientists/ML practitioners in designing fair ML pipelines.
This paper takes the first step in exploring this area by undertaking an
extensive empirical study comprising 60 combinations of interventions, 9
fairness metrics, 2 utility metrics (Accuracy and F1 Score) across 4 benchmark
datasets. We quantitatively analyze the experimental data to measure the impact
of multiple interventions on fairness, utility and population groups. We found
that applying multiple interventions results in better fairness and lower
utility than individual interventions on aggregate. However, adding more
interventions do no always result in better fairness or worse utility. The
likelihood of achieving high performance (F1 Score) along with high fairness
increases with larger number of interventions. On the downside, we found that
fairness-enhancing interventions can negatively impact different population
groups, especially the privileged group. This study highlights the need for new
fairness metrics that account for the impact on different population groups
apart from just the disparity between groups. Lastly, we offer a list of
combinations of interventions that perform best for different fairness and
utility metrics to aid the design of fair ML pipelines.

本研究着重探索不同阶段的机器学习管道中多个公正增强干预措施的累积效应对公正的影响，发现通过应用多个干预措施可提高公正性并降低效用度，但是增加更多的干预措施并不总是带来更好的公正性或更差的效用度，并提出了在不同公正和效用度指标下执行最佳干预组合以帮助设计公正的机器学习管道的清单。

级联去偏：研究多项增进公平的干预的累积效应

Cascaded Debiasing: Studying the Cumulative Effect of Multiple Fairness-Enhancing Interventions

Many statistical experiments involve comparing multiple population groups.
For example, a public opinion poll may ask which of several political
candidates commands the most support; a social scientific survey may report the
most common of several responses to a question; or, a clinical trial may
compare binary patient outcomes under several treatment conditions to determine
the most effective treatment. Having observed the "winner" (largest observed
response) in a noisy experiment, it is natural to ask whether that candidate,
survey response, or treatment is actually the "best" (stochastically largest
response). This article concerns the problem of rank verification --- post hoc
significance tests of whether the orderings discovered in the data reflect the
population ranks. For exponential family models, we show under mild conditions
that an unadjusted two-tailed pairwise test comparing the top two observations
(i.e., comparing the "winner" to the "runner-up") is a valid test of whether
the winner is truly the best. We extend our analysis to provide equally simple
procedures to obtain lower confidence bounds on the gap between the winning
population and the others, and to verify ranks beyond the first.

本文探讨了排名验证的问题，给出了针对指数族模型的简单方法，包括使用未调整的双尾成对检验来检验赢家是否真的是最好的，并进一步提供了获得差距置信下界的过程以及验证其他排名的方法。