Machine Reading Comprehension (MRC) is an essential task in evaluating natural language understanding. Existing MRC datasets primarily assess specific aspects of reading comprehension (RC), lacking a comprehensive MRC benchmark. To fill this gap, we first introduce a novel taxonomy that categorizes the key capabilities required for RC. Based on this taxonomy, we construct MRCEval, an MRC benchmark that leverages advanced Large Language Models (LLMs) as both sample generators and selection judges. MRCEval is a comprehensive, challenging and accessible benchmark designed to assess the RC capabilities of LLMs thoroughly, covering 13 distinct RC skills with a total of 2.1K high-quality multi-choice questions. We perform an extensive evaluation of 28 widely used open-source and proprietary models, highlighting that MRC continues to present significant challenges even in the era of LLMs.

本研究解决了现有机器阅读理解（MRC）数据集仅评估特定方面的阅读理解能力这一问题，提出了新的分类法以区分关键能力。基于该分类法，我们构建了MRCEval基准，全面评估了大型语言模型的阅读理解能力，发现即使在大型语言模型时代，MRC仍然面临重大挑战。

MRCEval：一个全面、具有挑战性和易于访问的机器阅读理解基准