As public sector agencies rapidly introduce new AI tools in high-stakes
domains like social services, it becomes critical to understand how decisions
to adopt these tools are made in practice. We borrow from the anthropological
practice to ``study up'' those in positions of power, and reorient our study of
public sector AI around those who have the power and responsibility to make
decisions about the role that AI tools will play in their agency. Through
semi-structured interviews and design activities with 16 agency
decision-makers, we examine how decisions about AI design and adoption are
influenced by their interactions with and assumptions about other actors within
these agencies (e.g., frontline workers and agency leaders), as well as those
above (legal systems and contracted companies), and below (impacted
communities). By centering these networks of power relations, our findings shed
light on how infrastructural, legal, and social factors create barriers and
disincentives to the involvement of a broader range of stakeholders in
decisions about AI design and adoption. Agency decision-makers desired more
practical support for stakeholder involvement around public sector AI to help
overcome the knowledge and power differentials they perceived between them and
other stakeholders (e.g., frontline workers and impacted community members).
Building on these findings, we discuss implications for future research and
policy around actualizing participatory AI approaches in public sector
contexts.

公共部门人工智能的设计和采用受到他们与其他干系人（如一线工作人员和受影响社区）、法律系统和承包公司之间相互作用及假设的影响，推动了对广泛利益相关者参与其设计和采用决策的障碍和抑制因素，决策者渴望更多实际支持以克服他们与其他利益相关者之间的知识和权力差异，研究和政策需重视在公共部门背景中实现参与式人工智能方法的推动。

研究公共部门人工智能：权力关系网络如何影响机构在人工智能设计和使用方面的决策

Studying Up Public Sector AI: How Networks of Power Relations Shape  Agency Decisions Around AI Design and Use

The rapid expansion of Learning Analytics (LA) and Artificial Intelligence in
Education (AIED) offers new scalable, data-intensive systems but also raises
concerns about data privacy and agency. Excluding stakeholders -- like students
and teachers -- from the design process can potentially lead to mistrust and
inadequately aligned tools. Despite a shift towards human-centred design in
recent LA and AIED research, there remain gaps in our understanding of the
importance of human control, safety, reliability, and trustworthiness in the
design and implementation of these systems. We conducted a systematic
literature review to explore these concerns and gaps. We analysed 108 papers to
provide insights about i) the current state of human-centred LA/AIED research;
ii) the extent to which educational stakeholders have contributed to the design
process of human-centred LA/AIED systems; iii) the current balance between
human control and computer automation of such systems; and iv) the extent to
which safety, reliability and trustworthiness have been considered in the
literature. Results indicate some consideration of human control in LA/AIED
system design, but limited end-user involvement in actual design. Based on
these findings, we recommend: 1) carefully balancing stakeholders' involvement
in designing and deploying LA/AIED systems throughout all design phases, 2)
actively involving target end-users, especially students, to delineate the
balance between human control and automation, and 3) exploring safety,
reliability, and trustworthiness as principles in future human-centred LA/AIED
systems.

通过系统文献综述，本研究揭示了在基于人性化设计的学习分析和人工智能教育系统中，人性化设计的现状以及教育利益相关者对设计过程的参与程度、人机控制后平衡、以及安全可靠性和可信度的考虑程度，并在此基础上提出了设计和部署这些系统时需要在各个阶段谨慎平衡利益相关者参与、积极吸纳最终用户，尤其是学生的参与，以及在未来的人性化学习分析和人工智能教育系统中探索安全、可靠性和可信度原则的建议。