We present JEC-QA, the largest question answering dataset in the legal
domain, collected from the National Judicial Examination of China. The
examination is a comprehensive evaluation of professional skills for legal
practitioners. College students are required to pass the examination to be
certified as a lawyer or a judge. The dataset is challenging for existing
question answering methods, because both retrieving relevant materials and
answering questions require the ability of logic reasoning. Due to the high
demand of multiple reasoning abilities to answer legal questions, the
state-of-the-art models can only achieve about 28% accuracy on JEC-QA, while
skilled humans and unskilled humans can reach 81% and 64% accuracy
respectively, which indicates a huge gap between humans and machines on this
task. We will release JEC-QA and our baselines to help improve the reasoning
ability of machine comprehension models. You can access the dataset from
this http URL

我们制作了 JEC-QA，这是中国国家司法考试中收集的最大的法律领域问答数据集，它要求逻辑推理能力用以解答问题。我们已经发布了该数据集和我们的基线模型，以帮助提高机器理解模型的推理能力。

JEC-QA：一个法律领域的问答数据集

JEC-QA: A Legal-Domain Question Answering Dataset

Recently, open-domain question answering (QA) has been combined with machine
comprehension models to find answers in a large knowledge source. As
open-domain QA requires retrieving relevant documents from text corpora to
answer questions, its performance largely depends on the performance of
document retrievers. However, since traditional information retrieval systems
are not effective in obtaining documents with a high probability of containing
answers, they lower the performance of QA systems. Simply extracting more
documents increases the number of irrelevant documents, which also degrades the
performance of QA systems. In this paper, we introduce Paragraph Ranker which
ranks paragraphs of retrieved documents for a higher answer recall with less
noise. We show that ranking paragraphs and aggregating answers using Paragraph
Ranker improves performance of open-domain QA pipeline on the four open-domain
QA datasets by 7.8% on average.

本文讲述了一种称为段落排名器的方法，它提高了开放域 QA 管道的性能，通过排名检索文档的段落以获取更高的答案回忆率和更少的噪声。在四个开放域 QA 数据集上，使用段落排名器排名段落和聚合答案相比未使用该方法，性能平均提高了 7.8%，也展示了开放域 QA 与机器理解模型的结合。