Algorithmic risk assessments are increasingly used to help humans make
decisions in high-stakes settings, such as medicine, criminal justice and
education. In each of these cases, the purpose of the risk assessment tool is
to inform actions, such as medical treatments or release conditions, often with
the aim of reducing the likelihood of an adverse event such as hospital
readmission or recidivism. Problematically, most tools are trained and
evaluated on historical data in which the outcomes observed depend on the
historical decision-making policy. These tools thus reflect risk under the
historical policy, rather than under the different decision options that the
tool is intended to inform. Even when tools are constructed to predict risk
under a specific decision, they are often improperly evaluated as predictors of
the target outcome.
Focusing on the evaluation task, in this paper we define counterfactual
analogues of common predictive performance and algorithmic fairness metrics
that we argue are better suited for the decision-making context. We introduce a
new method for estimating the proposed metrics using doubly robust estimation.
We provide theoretical results that show that only under strong conditions can
fairness according to the standard metric and the counterfactual metric
simultaneously hold. Consequently, fairness-promoting methods that target
parity in a standard fairness metric may --- and as we show empirically, do ---
induce greater imbalance in the counterfactual analogue. We provide empirical
comparisons on both synthetic data and a real world child welfare dataset to
demonstrate how the proposed method improves upon standard practice.

本文提出了针对算法风险评估工具的反事实度量方法，并使用双重稳健估计方法估算反事实度量。作者提出的度量方法更适合在决策上下文中使用；因此，纵观标准公平度量和反事实度量，两者仅在特定条件下才能同时成立，本文提供了理论结果和实证结果以证明他们的方法优于标准做法。