With an evergrowing number of LLMs reporting superlative performance for
English, their ability to perform equitably for different dialects of English
(i.e., dialect robustness) needs to be ascertained. Specifically, we use
English language (US English or Indian English) conversations between humans
who play the word-guessing game of `taboo'. We formulate two evaluative tasks:
target word prediction (TWP) (i.e.predict the masked target word in a
conversation) and target word selection (TWS) (i.e., select the most likely
masked target word in a conversation, from among a set of candidate words).
Extending MD3, an existing dialectic dataset of taboo-playing conversations, we
introduce M-MD3, a target-word-masked version of MD3 with the USEng and IndEng
subsets. We add two subsets: AITrans (where dialectic information is removed
from IndEng) and AIGen (where LLMs are prompted to generate conversations). Our
evaluation uses pre-trained and fine-tuned versions of two closed-source
(GPT-4/3.5) and two open-source LLMs (Mistral and Gemma). LLMs perform
significantly better for US English than Indian English for both TWP and TWS,
for all settings. While GPT-based models perform the best, the comparatively
smaller models work more equitably for short conversations (<8 turns). Our
results on AIGen and AITrans (the best and worst-performing subset)
respectively show that LLMs may learn a dialect of their own based on the
composition of the training data, and that dialect robustness is indeed a
challenging task. Our evaluation methodology exhibits a novel way to examine
attributes of language models using pre-existing dialogue datasets.

增强学习模型（LLMs）在不同英语方言上的表现以及预测和选择目标词的任务的采用的方法与结果

通过对话理解评估语言模型的方言鲁棒性

Evaluating Dialect Robustness of Language Models via Conversation  Understanding

Dialect differences caused by regional, social, and economic barriers cause
performance discrepancies for many groups of users of language technology.
Fair, inclusive, and equitable language technology must critically be dialect
invariant, meaning that performance remains constant over dialectal shifts.
Current English systems often fall significantly short of this ideal since they
are designed and tested on a single dialect: Standard American English. We
introduce Multi-VALUE -- a suite of resources for evaluating and achieving
English dialect invariance. We build a controllable rule-based translation
system spanning 50 English dialects and a total of 189 unique linguistic
features. Our translation maps Standard American English text to synthetic form
of each dialect, which uses an upper-bound on the natural density of features
in that dialect. First, we use this system to build stress tests for question
answering, machine translation, and semantic parsing tasks. Stress tests reveal
significant performance disparities for leading models on non-standard
dialects. Second, we use this system as a data augmentation technique to
improve the dialect robustness of existing systems. Finally, we partner with
native speakers of Chicano and Indian English to release new gold-standard
variants of the popular CoQA task.

介绍 Multi-VALUE—— 一种用于评估和实现英语方言不变性的资源套件，通过构建适用于 50 种英语方言的可控规则翻译系统，发现非标准方言上的主流模型存在显著性能差异，并以此作为数据增强技术来提高现有系统的方言稳健性，最后与 “Chicano” 和 “Indian English” 的母语使用者合作，发布了新的 CoQA 任务的标准变体。