To enable building and testing models on long-document comprehension, we
introduce QuALITY, a multiple-choice QA dataset with context passages in
English that have an average length of about 5,000 tokens, much longer than
typical current models can process. Unlike in prior work with passages, our
questions are written and validated by contributors who have read the entire
passage, rather than relying on summaries or excerpts. In addition, only half
of the questions are answerable by annotators working under tight time
constraints, indicating that skimming and simple search are not enough to
consistently perform well. Our baseline models perform poorly on this task
(55.4%) and significantly lag behind human performance (93.5%).

本文介绍了一个名为 QuALITY 的多项选择问答数据集，它包含英文上下文段落，其平均长度约为 5,000 个标记，远长于现有典型模型所能处理的长度。该数据集基于整篇文章编写和验证问题，有一半的问题无法在时间限制下回答，说明简单的浏览和搜索并不足以表现出优良的效果。作者的基准模型在这个任务上的表现较差（55.4％），并且显著落后于人类的表现（93.5％）。