Machine Reading Comprehension (MRC) is the task of answering a question over
a paragraph of text. While neural MRC systems gain popularity and achieve
noticeable performance, issues are being raised with the methodology used to
establish their performance, particularly concerning the data design of gold
standards that are used to evaluate them. There is but a limited understanding
of the challenges present in this data, which makes it hard to draw comparisons
and formulate reliable hypotheses. As a first step towards alleviating the
problem, this paper proposes a unifying framework to systematically investigate
the present linguistic features, required reasoning and background knowledge
and factual correctness on one hand, and the presence of lexical cues as a
lower bound for the requirement of understanding on the other hand. We propose
a qualitative annotation schema for the first and a set of approximative
metrics for the latter. In a first application of the framework, we analyse
modern MRC gold standards and present our findings: the absence of features
that contribute towards lexical ambiguity, the varying factual correctness of
the expected answers and the presence of lexical cues, all of which potentially
lower the reading comprehension complexity and quality of the evaluation data.

本论文提出了一个统一框架来系统研究现有的语言特征、推理需求、背景知识和事实正确性以及词汇线索的存在。研究结果发现 MRC 数据存在词汇歧义、答案事实正确性的差异和词汇线索的存在，这些都可能降低阅读理解的复杂性和质量。