We present the ARC-DA dataset, a direct-answer ("open response", "freeform")
version of the ARC (AI2 Reasoning Challenge) multiple-choice dataset. While ARC
has been influential in the community, its multiple-choice format is
unrepresentative of real-world questions, and multiple choice formats can be
particularly susceptible to artifacts. The ARC-DA dataset addresses these
concerns by converting questions to direct-answer format using a combination of
crowdsourcing and expert review. The resulting dataset contains 2985 questions
with a total of 8436 valid answers (questions typically have more than one
valid answer). ARC-DA is one of the first DA datasets of natural questions that
often require reasoning, and where appropriate question decompositions are not
evident from the questions themselves. We describe the conversion approach
taken, appropriate evaluation metrics, and several strong models. Although
high, the best scores (81% GENIE, 61.4% F1, 63.2% ROUGE-L) still leave
considerable room for improvement. In addition, the dataset provides a natural
setting for new research on explanation, as many questions require reasoning to
construct answers. We hope the dataset spurs further advances in complex
question-answering by the community. ARC-DA is available at
this https URL

ARC-DA 数据集通过众包与专家审查，将 ARC 多项选择数据集转化为直接回答形式。该数据集包含 2985 个问题，其中至少有 8436 个有效答案，并在自然问题推理中寻找了新的研究方向。