In AI-assisted decision-making, a central promise of putting a human in the
loop is that they should be able to complement the AI system by adhering to its
correct and overriding its mistaken recommendations. In practice, however, we
often see that humans tend to over- or under-rely on AI recommendations,
meaning that they either adhere to wrong or override correct recommendations.
Such reliance behavior is detrimental to decision-making accuracy. In this
work, we articulate and analyze the interdependence between reliance behavior
and accuracy in AI-assisted decision-making, which has been largely neglected
in prior work. We also propose a visual framework to make this interdependence
more tangible. This framework helps us interpret and compare empirical
findings, as well as obtain a nuanced understanding of the effects of
interventions (e.g., explanations) in AI-assisted decision-making. Finally, we
infer several interesting properties from the framework: (i) when humans
under-rely on AI recommendations, there may be no possibility for them to
complement the AI in terms of decision-making accuracy; (ii) when humans cannot
discern correct and wrong AI recommendations, no such improvement can be
expected either; (iii) interventions may lead to an increase in decision-making
accuracy that is solely driven by an increase in humans' adherence to AI
recommendations, without any ability to discern correct and wrong. Our work
emphasizes the importance of measuring and reporting both effects on accuracy
and reliance behavior when empirically assessing interventions.

本文分析了 AI 辅助决策中人类对模型建议过度或不足的倾向对准确性的影响，并提出了一个可视化框架以帮助比较和解释实证结果和干预策略。

AI 辅助决策中依赖行为和准确性的相互依存性

On the Interdependence of Reliance Behavior and Accuracy in AI-Assisted  Decision-Making

Proponents of explainable AI have often argued that it constitutes an
essential path towards algorithmic fairness. Prior works examining these claims
have primarily evaluated explanations based on their effects on humans'
perceptions, but there is scant research on the relationship between
explanations and distributive fairness of AI-assisted decisions. In this paper,
we conduct an empirical study to examine the relationship between feature-based
explanations and distributive fairness, mediated by human perceptions and
reliance on AI recommendations. Our findings show that explanations influence
fairness perceptions, which, in turn, relate to humans' tendency to adhere to
AI recommendations. However, our findings suggest that such explanations do not
enable humans to discern correct and wrong AI recommendations. Instead, we show
that they may affect reliance irrespective of the correctness of AI
recommendations. Depending on which features an explanation highlights, this
can foster or hinder distributive fairness: when explanations highlight
features that are task-irrelevant and evidently associated with the sensitive
attribute, this prompts overrides that counter stereotype-aligned AI
recommendations. Meanwhile, if explanations appear task-relevant, this induces
reliance behavior that reinforces stereotype-aligned errors. These results show
that feature-based explanations are not a reliable mechanism to improve
distributive fairness, as their ability to do so relies on a human-in-the-loop
operationalization of the flawed notion of "fairness through unawareness".
Finally, our study design provides a blueprint to evaluate the suitability of
other explanations as pathways towards improved distributive fairness of
AI-assisted decisions.

研究了基于 AI 解释和分布式公平性之间的关系，发现解释会影响公平感知，从而与人类对 AI 建议的依赖关系发生关联。该研究表明，基于特征的解释并不是改善分布式公正性的可靠机制。