English Natural Language Understanding (NLU) systems have achieved great
performances and even outperformed humans on benchmarks like GLUE and
SuperGLUE. However, these benchmarks contain only textbook Standard American
English (SAE). Other dialects have been largely overlooked in the NLP
community. This leads to biased and inequitable NLU systems that serve only a
sub-population of speakers. To understand disparities in current models and to
facilitate more dialect-competent NLU systems, we introduce the VernAcular
Language Understanding Evaluation (VALUE) benchmark, a challenging variant of
GLUE that we created with a set of lexical and morphosyntactic transformation
rules. In this initial release (V.1), we construct rules for 11 features of
African American Vernacular English (AAVE), and we recruit fluent AAVE speakers
to validate each feature transformation via linguistic acceptability judgments
in a participatory design manner. Experiments show that these new dialectal
features can lead to a drop in model performance. To run the transformation
code and download both synthetic and gold-standard dialectal GLUE benchmarks,
see this https URL

通过创建一个有挑战性的 GLUE 变体（VernAcular Language Understanding Evaluation (VALUE) benchmark），该研究介绍了一组词汇和形态变换规则，构造为 African American Vernacular English（AAVE） 的 11 个特征。实验表明这些新的方言特征可能会导致模型性能下降。

了解自然语言理解中的方言差异

VALUE: Understanding Dialect Disparity in NLU

The VALUE (Video-And-Language Understanding Evaluation) benchmark is newly
introduced to evaluate and analyze multi-modal representation learning
algorithms on three video-and-language tasks: Retrieval, QA, and Captioning.
The main objective of the VALUE challenge is to train a task-agnostic model
that is simultaneously applicable for various tasks with different
characteristics. This technical report describes our winning strategies for the
VALUE challenge: 1) single model optimization, 2) transfer learning with visual
concepts, and 3) task-aware ensemble. The first and third strategies are
designed to address heterogeneous characteristics of each task, and the second
one is to leverage rich and fine-grained visual information. We provide a
detailed and comprehensive analysis with extensive experimental results. Based
on our approach, we ranked first place on the VALUE and QA phases for the
competition.

本技术报告介绍了我们在 VALUE 挑战赛中的优胜策略：单一模型优化，使用视觉概念的迁移学习以及任务感知的集成。根据我们的方法，我们在比赛的 VALUE 和 QA 阶段中排名第一。