Benchmarking is seen as critical to assessing progress in NLP. However,
creating a benchmark involves many design decisions (e.g., which datasets to
include, which metrics to use) that often rely on tacit, untested assumptions
about what the benchmark is intended to measure or is actually measuring. There
is currently no principled way of analyzing these decisions and how they impact
the validity of the benchmark's measurements. To address this gap, we draw on
evidence-centered design in educational assessments and propose
Evidence-Centered Benchmark Design (ECBD), a framework which formalizes the
benchmark design process into five modules. ECBD specifies the role each module
plays in helping practitioners collect evidence about capabilities of interest.
Specifically, each module requires benchmark designers to describe, justify,
and support benchmark design choices -- e.g., clearly specifying the
capabilities the benchmark aims to measure or how evidence about those
capabilities is collected from model responses. To demonstrate the use of ECBD,
we conduct case studies with three benchmarks: BoolQ, SuperGLUE, and HELM. Our
analysis reveals common trends in benchmark design and documentation that could
threaten the validity of benchmarks' measurements.

综合证据为中心的基准设计（ECBD）框架确定角色，以帮助从案例研究中收集与能力相关的证据，同时发现了基准设计和文档编写的常见趋势可能会威胁基准测试的有效性。

ECBD: NLP 的证据中心基准设计

ECBD: Evidence-Centered Benchmark Design for NLP

Despite their success, large pre-trained multilingual models have not
completely alleviated the need for labeled data, which is cumbersome to collect
for all target languages. Zero-shot cross-lingual transfer is emerging as a
practical solution: pre-trained models later fine-tuned on one transfer
language exhibit surprising performance when tested on many target languages.
English is the dominant source language for transfer, as reinforced by popular
zero-shot benchmarks. However, this default choice has not been systematically
vetted. In our study, we compare English against other transfer languages for
fine-tuning, on two pre-trained multilingual models (mBERT and mT5) and
multiple classification and question answering tasks. We find that other
high-resource languages such as German and Russian often transfer more
effectively, especially when the set of target languages is diverse or unknown
a priori. Unexpectedly, this can be true even when the training sets were
automatically translated from English. This finding can have immediate impact
on multilingual zero-shot systems, and should inform future benchmark designs.

本文对于使用不同语言进行零样本跨语言转移的多语言模型进行了研究，发现高资源语言如德语和俄语在多数情况下能够更加有效地进行迁移学习，即使训练集是自动从英语翻译而来。同时，该结论对于多语言零样本系统有着重要影响，并且应该指导未来的基准设计。