We evaluate how well LLMs understand African American Language (AAL) in
comparison to their performance on White Mainstream English (WME), the
encouraged "standard" form of English taught in American classrooms. We measure
LLM performance using automatic metrics and human judgments for two tasks: a
counterpart generation task, where a model generates AAL (or WME) given WME (or
AAL), and a masked span prediction (MSP) task, where models predict a phrase
that was removed from their input. Our contributions include: (1) evaluation of
six pre-trained, large language models on the two language generation tasks;
(2) a novel dataset of AAL text from multiple contexts (social media, hip-hop
lyrics, focus groups, and linguistic interviews) with human-annotated
counterparts in WME; and (3) documentation of model performance gaps that
suggest bias and identification of trends in lack of understanding of AAL
features.

该研究评估 LLM 在理解非洲裔美国人语言（AAL）方面的能力，比较其在白人主流英语（WME）方面的表现。通过两项任务，评估 LLM 的表现并发现了模型性能差距、偏见和 AAL 特征的理解欠缺。