A critical component when developing question-answering AIs is an adversarial
dataset that challenges models to adapt to the complex syntax and reasoning
underlying our natural language. Present techniques for procedurally generating
adversarial texts are not robust enough for training on complex tasks such as
answering multi-sentence trivia questions. We instead turn to human-generated
data by introducing an interface for collecting adversarial human-written
trivia questions. Our interface is aimed towards question writers and players
of Quiz Bowl, a buzzer-based trivia competition where paragraph-long questions
consist of a sequence of clues of decreasing difficulty. To incentivize usage,
a suite of machine learning-based tools in our interface assist humans in
writing questions that are more challenging to answer for Quiz Bowl players and
computers alike. Not only does our interface gather training data for the
groundbreaking Quiz Bowl AI project QANTA, but it is also a proof-of-concept of
future adversarial data collection for question-answering systems. The results
of performance-testing our interface with ten originally-composed questions
indicate that, despite some flaws, our interface's novel question-writing
features as well as its real-time exposure of useful responses from our machine
models could facilitate and enhance the collection of adversarial questions.

通过引入一种收集对抗性人工编写的问题的界面，我们利用人类生成的数据来开发让机器更好应对复杂任务的对抗性数据集，这对于开发问答人工智能是至关重要的。我们的界面不仅为创新的 Quiz Bowl AI 项目 QANTA 收集训练数据，同时也是为问答系统未来对抗性数据收集的一个概念验证。

对抗性智力问答写作的新界面

A novel interface for adversarial trivia question-writing

We present our experience as annotators in the creation of high-quality,
adversarial machine-reading-comprehension data for extractive QA for Task 1 of
the First Workshop on Dynamic Adversarial Data Collection (DADC). DADC is an
emergent data collection paradigm with both models and humans in the loop. We
set up a quasi-experimental annotation design and perform quantitative analyses
across groups with different numbers of annotators focusing on successful
adversarial attacks, cost analysis, and annotator confidence correlation. We
further perform a qualitative analysis of our perceived difficulty of the task
given the different topics of the passages in our dataset and conclude with
recommendations and suggestions that might be of value to people working on
future DADC tasks and related annotation interfaces.

本文介绍我们在创建高质量的、针对任务 1 的机器阅读理解数据的过程中的经验，并描述了我们基于不同专题的定量定性分析结果以及与注释员信心的相关性，最后我们提出了相关建议。

利用人类和模型循环收集高质量对抗数据以用于机器阅读理解任务

Collecting high-quality adversarial data for machine reading comprehension tasks with humans and models in the loop

In adversarial data collection (ADC), a human workforce interacts with a
model in real time, attempting to produce examples that elicit incorrect
predictions. Researchers hope that models trained on these more challenging
datasets will rely less on superficial patterns, and thus be less brittle.
However, despite ADC's intuitive appeal, it remains unclear when training on
adversarial datasets produces more robust models. In this paper, we conduct a
large-scale controlled study focused on question answering, assigning workers
at random to compose questions either (i) adversarially (with a model in the
loop); or (ii) in the standard fashion (without a model). Across a variety of
models and datasets, we find that models trained on adversarial data usually
perform better on other adversarial datasets but worse on a diverse collection
of out-of-domain evaluation sets. Finally, we provide a qualitative analysis of
adversarial (vs standard) data, identifying key differences and offering
guidance for future research.

本文通过对问答模型进行大规模对比研究，发现对抗训练对提升模型在对抗数据集上的表现有帮助，但在领域外数据集上却不够稳健；同时，文章进行了对抗数据和标准数据的质量比较，为未来的研究提供指导。