We propose a test of fairness in score-based ranking systems called matched pair calibration. Our approach constructs a set of matched item pairs with minimal confounding differences between subgroups before computing an appropriate measure of ranking error over the set. The matching step ensures that we compare subgroup outcomes between identically scored items so that measured performance differences directly imply unfairness in subgroup-level exposures. We show how our approach generalizes the fairness intuitions of calibration from a binary classification setting to ranking and connect our approach to other proposals for ranking fairness measures. Moreover, our strategy shows how the logic of marginal outcome tests extends to cases where the analyst has access to model scores. Lastly, we provide an example of applying matched pair calibration to a real-word ranking data set to demonstrate its efficacy in detecting ranking bias.

我们提出了一种测试评分排序系统公平性的方法，称为匹配双校准。我们的方法构建了一组匹配的项目对，这些项目对子组之间的混杂差异最小，然后在集合中计算适当的排名误差度量，并确保比较相同评分的项目子组结果，以便测量绩效差异直接暗示着子组水平的不公平性。

排名公平性匹配对校准