We discuss the role of humans in algorithmic decision-making (ADM) for
socially relevant problems from a technical and philosophical perspective. In
particular, we illustrate tensions arising from diverse expectations, values,
and constraints by and on the humans involved. To this end, we assume that a
strategic decision-maker (SDM) introduces ADM to optimize strategic and
societal goals while the algorithms' recommended actions are overseen by a
practical decision-maker (PDM) - a specific human-in-the-loop - who makes the
final decisions. While the PDM is typically assumed to be a corrective, it can
counteract the realization of the SDM's desired goals and societal values not
least because of a misalignment of these values and unmet information needs of
the PDM. This has significant implications for the distribution of power
between the stakeholders in ADM, their constraints, and information needs. In
particular, we emphasize the overseeing PDM's role as a potential political and
ethical decision maker, who acts expected to balance strategic, value-driven
objectives and on-the-ground individual decisions and constraints. We
demonstrate empirically, on a machine learning benchmark dataset, the
significant impact an overseeing PDM's decisions can have even if the PDM is
constrained to performing only a limited amount of actions differing from the
algorithms' recommendations. To ensure that the SDM's intended values are
realized, the PDM needs to be provided with appropriate information conveyed
through tailored explanations and its role must be characterized clearly. Our
findings emphasize the need for an in-depth discussion of the role and power of
the PDM and challenge the often-taken view that just including a
human-in-the-loop in ADM ensures the 'correct' and 'ethical' functioning of the
system.

我们从技术和哲学的角度讨论人类在算法决策（ADM）中的作用，具体描述了涉及的不同期望、价值观和限制所引发的紧张关系，以及介入其中的人类所面临的挑战和影响。