For a natural language understanding benchmark to be useful in research, it
has to consist of examples that are diverse and difficult enough to
discriminate among current and near-future state-of-the-art systems. However,
we do not yet know how best to select text sources to collect a variety of
challenging examples. In this study, we crowdsource multiple-choice reading
comprehension questions for passages taken from seven qualitatively distinct
sources, analyzing what attributes of passages contribute to the difficulty and
question types of the collected examples. To our surprise, we find that passage
source, length, and readability measures do not significantly affect question
difficulty. Through our manual annotation of seven reasoning types, we observe
several trends between passage sources and reasoning types, e.g., logical
reasoning is more often required in questions written for technical passages.
These results suggest that when creating a new benchmark dataset, selecting a
diverse set of passages can help ensure a diverse range of question types, but
that passage difficulty need not be a priority.

通过众包多选阅读理解问题，分析来自七个质量不同的来源的段落的属性如何影响问题难度及类型，发现段落的来源、长度和可读性等因素并不影响问题的难度，而逻辑推理更常常出现在针对技术文章的问题中。因此，在创建新的基准数据集时，选择多样性的段落可以确保问题类型的多样性，而段落的难度则不是最重要的。