We present a novel method for obtaining high-quality, domain-targeted
multiple choice questions from crowd workers. Generating these questions can be
difficult without trading away originality, relevance or diversity in the
answer options. Our method addresses these problems by leveraging a large
corpus of domain-specific text and a small set of existing questions. It
produces model suggestions for document selection and answer distractor choice
which aid the human question generation process. With this method we have
assembled SciQ, a dataset of 13.7K multiple choice science exam questions
(Dataset available at this http URL). We demonstrate that the
method produces in-domain questions by providing an analysis of this new
dataset and by showing that humans cannot distinguish the crowdsourced
questions from original questions. When using SciQ as additional training data
to existing questions, we observe accuracy improvements on real science exams.

本文提出一种新的方法来从众包工作者那里获得高质量、针对领域的多项选择题，该方法通过利用大量的领域特定文本和一小部分现有问题，产生文档选择和答案干扰选项的模型建议，辅助人类提问生成过程。使用这种方法，我们已经组装了 13.7K 的多项选择科学考试题的 SciQ 数据集，并通过提供对这个新数据集的分析和展示人们无法区分众包问题与原问题，证明了这种方法可以产生领域内的问题。当将 SciQ 用作现有问题的附加训练数据时，在真实科学考试中观察到了准确度的提高。