Decision-making with information displays is a key focus of research in areas
like explainable AI, human-AI teaming, and data visualization. However, what
constitutes a decision problem, and what is required for an experiment to be
capable of concluding that human decisions are flawed in some way, remain open
to speculation. We present a widely applicable definition of a decision problem
synthesized from statistical decision theory and information economics. We
argue that to attribute loss in human performance to forms of bias, an
experiment must provide participants with the information that a rational agent
would need to identify the normative decision. We evaluate the extent to which
recent evaluations of decision-making from the literature on AI-assisted
decisions achieve this criteria. We find that only 6 (17\%) of 35 studies that
claim to identify biased behavior present participants with sufficient
information to characterize their behavior as deviating from good
decision-making. We motivate the value of studying well-defined decision
problems by describing a characterization of performance losses they allow us
to conceive. In contrast, the ambiguities of a poorly communicated decision
problem preclude normative interpretation. We conclude with recommendations for
practice.

决策与信息展示是可解释人工智能、人工智能与人类协作以及数据可视化等领域的研究重点。然而，何为决策问题以及为何需要实验来得出人类决策存在某种缺陷的结论，这些问题仍然存在争议。我们提出了一种广泛适用的决策问题定义，结合了统计决策理论和信息经济学。我们认为，要将人类表现中的损失归因于偏见形式，实验必须向参与者提供一个理性代理人所需的信息，以识别规范决策。我们评估了近期关于人工智能辅助决策的决策研究评估在多大程度上满足这一标准。我们发现，在 35 项声称发现存在偏见行为的研究中，只有 6 项（17%）为参与者提供足够的信息来表征其行为与良好决策制定有所偏离。通过描述性能损失的特征，我们阐述了研究明确定义的决策问题的价值，并对问题表述不清晰的决策问题的模糊性进行了对比，这限制了规范的解释。最后，我们给出了一些建议以促进实践。

评估人类决策的实验的决策理论基础

Decision Theoretic Foundations for Experiments Evaluating Human  Decisions

Recent works have shown considerable improvements in task-oriented dialogue
(TOD) systems by utilizing pretrained large language models (LLMs) in an
end-to-end manner. However, the biased behavior of each component in a TOD
system and the error propagation issue in the end-to-end framework can lead to
seriously biased TOD responses. Existing works of fairness only focus on the
total bias of a system. In this paper, we propose a diagnosis method to
attribute bias to each component of a TOD system. With the proposed attribution
method, we can gain a deeper understanding of the sources of bias.
Additionally, researchers can mitigate biased model behavior at a more granular
level. We conduct experiments to attribute the TOD system's bias toward three
demographic axes: gender, age, and race. Experimental results show that the
bias of a TOD system usually comes from the response generation model.

使用预训练的大型语言模型，本文提出了一种诊断方法来确定任务导向对话系统中每个组件的偏见，并通过实验结果表明这种偏见主要来自回应生成模型。