Disaggregated evaluation is a central task in AI fairness assessment, with
the goal to measure an AI system's performance across different subgroups
defined by combinations of demographic or other sensitive attributes. The
standard approach is to stratify the evaluation data across subgroups and
compute performance metrics separately for each group. However, even for
moderately-sized evaluation datasets, sample sizes quickly get small once
considering intersectional subgroups, which greatly limits the extent to which
intersectional groups are considered in many disaggregated evaluations. In this
work, we introduce a structured regression approach to disaggregated evaluation
that we demonstrate can yield reliable system performance estimates even for
very small subgroups. We also provide corresponding inference strategies for
constructing confidence intervals and explore how goodness-of-fit testing can
yield insight into the structure of fairness-related harms experienced by
intersectional groups. We evaluate our approach on two publicly available
datasets, and several variants of semi-synthetic data. The results show that
our method is considerably more accurate than the standard approach, especially
for small subgroups, and goodness-of-fit testing helps identify the key factors
that drive differences in performance.

通过引入结构回归方法进行细分评估，以演示即使对于非常小的子群体，也可以得出可靠的系统性能估计，进而提供相应的推断策略以构建置信区间，并探索拟合度检验如何揭示交叉群体所经历的公平相关伤害的结构。

基于结构回归的模型性能评估方法在交叉子群中的应用

A structured regression approach for evaluating model performance across  intersectional subgroups

In a well-calibrated risk prediction model, the average predicted probability
is close to the true event rate for any given subgroup. Such models are
reliable across heterogeneous populations and satisfy strong notions of
algorithmic fairness. However, the task of auditing a model for strong
calibration is well-known to be difficult -- particularly for machine learning
(ML) algorithms -- due to the sheer number of potential subgroups. As such,
common practice is to only assess calibration with respect to a few predefined
subgroups. Recent developments in goodness-of-fit testing offer potential
solutions but are not designed for settings with weak signal or where the
poorly calibrated subgroup is small, as they either overly subdivide the data
or fail to divide the data at all. We introduce a new testing procedure based
on the following insight: if we can reorder observations by their expected
residuals, there should be a change in the association between the predicted
and observed residuals along this sequence if a poorly calibrated subgroup
exists. This lets us reframe the problem of calibration testing into one of
changepoint detection, for which powerful methods already exist. We begin with
introducing a sample-splitting procedure where a portion of the data is used to
train a suite of candidate models for predicting the residual, and the
remaining data are used to perform a score-based cumulative sum (CUSUM) test.
To further improve power, we then extend this adaptive CUSUM test to
incorporate cross-validation, while maintaining Type I error control under
minimal assumptions. Compared to existing methods, the proposed procedure
consistently achieved higher power in simulation studies and more than doubled
the power when auditing a mortality risk prediction model.

通过改进的测试程序，我们引入了一种新的基于改变点检测的校准测试方法，可以有效评估风险预测模型的准确性和算法的公平性。

该模型对所有人都可靠吗？强校准测试

Is this model reliable for everyone? Testing for strong calibration

Computable Stein discrepancies have been deployed for a variety of
applications, ranging from sampler selection in posterior inference to
approximate Bayesian inference to goodness-of-fit testing. Existing
convergence-determining Stein discrepancies admit strong theoretical guarantees
but suffer from a computational cost that grows quadratically in the sample
size. While linear-time Stein discrepancies have been proposed for
goodness-of-fit testing, they exhibit avoidable degradations in testing power
-- even when power is explicitly optimized. To address these shortcomings, we
introduce feature Stein discrepancies ($\Phi$SDs), a new family of quality
measures that can be cheaply approximated using importance sampling. We show
how to construct $\Phi$SDs that provably determine the convergence of a sample
to its target and develop high-accuracy approximations -- random $\Phi$SDs
(R$\Phi$SDs) -- which are computable in near-linear time. In our experiments
with sampler selection for approximate posterior inference and goodness-of-fit
testing, R$\Phi$SDs perform as well or better than quadratic-time KSDs while
being orders of magnitude faster to compute.

提出了一种新的质量度量方法 - 特征 Stein 差异度量（PhiSD），可通过重要抽样方法进行便宜的近线性逼近，适用于近似后验推断和适配度检验。在实验证明，在速度方面比平方时间差异度量 (KSD) 更快，可在计算中进行价值代替。