The recent work of Clark et al. introduces the AI2 Reasoning Challenge (ARC)
and the associated ARC dataset that partitions open domain, complex science
questions into an Easy Set and a Challenge Set. That paper includes an analysis
of 100 questions with respect to the types of knowledge and reasoning required
to answer them; however, it does not include clear definitions of these types,
nor does it offer information about the quality of the labels. We propose a
comprehensive set of definitions of knowledge and reasoning types necessary for
answering the questions in the ARC dataset. Using ten annotators and a
sophisticated annotation interface, we analyze the distribution of labels
across the Challenge Set and statistics related to them. Additionally, we
demonstrate that although naive information retrieval methods return sentences
that are irrelevant to answering the query, sufficient supporting text is often
present in the (ARC) corpus. Evaluating with human-selected relevant sentences
improves the performance of a neural machine comprehension model by 42 points.

本文提出了一套完整的科学问题解答所需的知识和推理类型定义方法，回答 ARC 数据集中的问题需要的标注信息分布和相关统计的分析，证明了人工选取的相关句子能够提高神经机器理解模型的 42 个百分点的性能。