Readers of academic research papers often read with the goal of answering
specific questions. Question Answering systems that can answer those questions
can make consumption of the content much more efficient. However, building such
tools requires data that reflect the difficulty of the task arising from
complex reasoning about claims made in multiple parts of a paper. In contrast,
existing information-seeking question answering datasets usually contain
questions about generic factoid-type information. We therefore present QASPER,
a dataset of 5,049 questions over 1,585 Natural Language Processing papers.
Each question is written by an NLP practitioner who read only the title and
abstract of the corresponding paper, and the question seeks information present
in the full text. The questions are then answered by a separate set of NLP
practitioners who also provide supporting evidence to answers. We find that
existing models that do well on other QA tasks do not perform well on answering
these questions, underperforming humans by at least 27 F1 points when answering
them from entire papers, motivating further research in document-grounded,
information-seeking QA, which our dataset is designed to facilitate.

本文介绍了 QASPER（一个基于自然语言处理论文的问题数据集），旨在提醒需要收集反映多部分论文主张的复杂推理难度的数据，以利于构建支持文档查找的问答系统。