From the social sciences to machine learning, it has been well documented
that metrics to be optimized are not always aligned with social welfare. In
healthcare, Dranove et al. [12] showed that publishing surgery mortality
metrics actually harmed the welfare of sicker patients by increasing provider
selection behavior. Using a principal-agent model, we directly study the
incentive misalignments that arise from such average treated outcome metrics,
and show that the incentives driving treatment decisions would align with
maximizing total patient welfare if the metrics (i) accounted for
counterfactual untreated outcomes and (ii) considered total welfare instead of
average welfare among treated patients. Operationalizing this, we show how
counterfactual metrics can be modified to satisfy desirable properties when
used for ranking. Extending to realistic settings when the providers observe
more about patients than the regulatory agencies do, we bound the decay in
performance by the degree of information asymmetry between the principal and
the agent. In doing so, our model connects principal-agent information
asymmetry with unobserved heterogeneity in causal inference.

该研究通过一个主体代理模型研究了治疗指标存在的激励错位问题并表明，如果治疗指标考虑了未治疗结果，同时综合考虑整体病人福利而非仅治疗病人的平均福利，就能最大化病人福利；此外，当治疗者观察到的病人信息比监管机构更多时，该模型连接了主体代理信息不对称和因果推断中未观测到的异质性。

操作化反事实度量：激励、排名和信息不对称

Operationalizing Counterfactual Metrics: Incentives, Ranking, and  Information Asymmetry

Algorithmic risk assessments are increasingly used to help humans make
decisions in high-stakes settings, such as medicine, criminal justice and
education. In each of these cases, the purpose of the risk assessment tool is
to inform actions, such as medical treatments or release conditions, often with
the aim of reducing the likelihood of an adverse event such as hospital
readmission or recidivism. Problematically, most tools are trained and
evaluated on historical data in which the outcomes observed depend on the
historical decision-making policy. These tools thus reflect risk under the
historical policy, rather than under the different decision options that the
tool is intended to inform. Even when tools are constructed to predict risk
under a specific decision, they are often improperly evaluated as predictors of
the target outcome.
Focusing on the evaluation task, in this paper we define counterfactual
analogues of common predictive performance and algorithmic fairness metrics
that we argue are better suited for the decision-making context. We introduce a
new method for estimating the proposed metrics using doubly robust estimation.
We provide theoretical results that show that only under strong conditions can
fairness according to the standard metric and the counterfactual metric
simultaneously hold. Consequently, fairness-promoting methods that target
parity in a standard fairness metric may --- and as we show empirically, do ---
induce greater imbalance in the counterfactual analogue. We provide empirical
comparisons on both synthetic data and a real world child welfare dataset to
demonstrate how the proposed method improves upon standard practice.

本文提出了针对算法风险评估工具的反事实度量方法，并使用双重稳健估计方法估算反事实度量。作者提出的度量方法更适合在决策上下文中使用；因此，纵观标准公平度量和反事实度量，两者仅在特定条件下才能同时成立，本文提供了理论结果和实证结果以证明他们的方法优于标准做法。