The rapid emergence of generative Language Models (LMs) has led to growing
concern about the impacts that their unexamined adoption may have on the social
well-being of diverse user groups. Meanwhile, LMs are increasingly being
adopted in K-20 schools and one-on-one student settings with minimal
investigation of potential harms associated with their deployment. Motivated in
part by real-world/everyday use cases (e.g., an AI writing assistant) this
paper explores the potential psychosocial harms of stories generated by five
leading LMs in response to open-ended prompting. We extend findings of
stereotyping harms analyzing a total of 150K 100-word stories related to
student classroom interactions. Examining patterns in LM-generated character
demographics and representational harms (i.e., erasure, subordination, and
stereotyping) we highlight particularly egregious vignettes, illustrating the
ways LM-generated outputs may influence the experiences of users with
marginalized and minoritized identities, and emphasizing the need for a
critical understanding of the psychosocial impacts of generative AI tools when
deployed and utilized in diverse social contexts.

该研究探讨引起关注的生成式语言模型在教育领域可能产生的心理社会危害，分析了与学生课堂互动相关的 15 万个 100 字的故事中生成式语言模型所产生的角色人口统计学和再现伤害，强调了生成式人工智能工具在多样化社会环境中部署和使用时对于具有边缘化和少数族裔身份的用户体验可能产生的心理社会影响的重要性。