We present our experience as annotators in the creation of high-quality,
adversarial machine-reading-comprehension data for extractive QA for Task 1 of
the First Workshop on Dynamic Adversarial Data Collection (DADC). DADC is an
emergent data collection paradigm with both models and humans in the loop. We
set up a quasi-experimental annotation design and perform quantitative analyses
across groups with different numbers of annotators focusing on successful
adversarial attacks, cost analysis, and annotator confidence correlation. We
further perform a qualitative analysis of our perceived difficulty of the task
given the different topics of the passages in our dataset and conclude with
recommendations and suggestions that might be of value to people working on
future DADC tasks and related annotation interfaces.

本文介绍我们在创建高质量的、针对任务 1 的机器阅读理解数据的过程中的经验，并描述了我们基于不同专题的定量定性分析结果以及与注释员信心的相关性，最后我们提出了相关建议。

利用人类和模型循环收集高质量对抗数据以用于机器阅读理解任务

Collecting high-quality adversarial data for machine reading comprehension tasks with humans and models in the loop

To create models that are robust across a wide range of test inputs, training
datasets should include diverse examples that span numerous phenomena. Dynamic
adversarial data collection (DADC), where annotators craft examples that
challenge continually improving models, holds promise as an approach for
generating such diverse training sets. Prior work has shown that running DADC
over 1-3 rounds can help models fix some error types, but it does not
necessarily lead to better generalization beyond adversarial test data. We
argue that running DADC over many rounds maximizes its training-time benefits,
as the different rounds can together cover many of the task-relevant phenomena.
We present the first study of longer-term DADC, where we collect 20 rounds of
NLI examples for a small set of premise paragraphs, with both adversarial and
non-adversarial approaches. Models trained on DADC examples make 26% fewer
errors on our expert-curated test set compared to models trained on
non-adversarial data. Our analysis shows that DADC yields examples that are
more difficult, more lexically and syntactically diverse, and contain fewer
annotation artifacts compared to non-adversarial examples.

使用动态对抗数据采集（DADC）能够不断地生成具有挑战性的例子，获得更多的训练样本，可提高模型泛化能力，同时减少模型的错误率。我们进行了长期的 DADC 研究，并使用对抗性和非对抗性方法收集了 20 个 NLI 示例，该方法训练的模型在我们的测试集上减少了 26％的错误率。