Over the last five years, research on Relation Extraction (RE) witnessed
extensive progress with many new dataset releases. At the same time, setup
clarity has decreased, contributing to increased difficulty of reliable
empirical evaluation (Taillé et al., 2020). In this paper, we provide a
comprehensive survey of RE datasets, and revisit the task definition and its
adoption by the community. We find that cross-dataset and cross-domain setups
are particularly lacking. We present an empirical study on scientific Relation
Classification across two datasets. Despite large data overlap, our analysis
reveals substantial discrepancies in annotation. Annotation discrepancies
strongly impact Relation Classification performance, explaining large drops in
cross-dataset evaluations. Variation within further sub-domains exists but
impacts Relation Classification only to limited degrees. Overall, our study
calls for more rigour in reporting setups in RE and evaluation across multiple
test sets.

本文对关系抽取数据集进行了全面的调查及研究任务定义及其应用。 研究发现，跨数据集和跨领域设置特别缺乏，并通过对两个数据集的科学关系分类进行了实证研究，发现注释存在实质性的差异，这解释了跨数据集评估中大幅度下降的原因。 最后，本研究呼吁在关系抽取中更加严格地报告设定和跨多个测试集进行评估。