Despite efforts to distinguish three different evaluation setups (Bekoulis et
al., 2018), numerous end-to-end Relation Extraction (RE) articles present
unreliable performance comparison to previous work. In this paper, we first
identify several patterns of invalid comparisons in published papers and
describe them to avoid their propagation. We then propose a small empirical
study to quantify the impact of the most common mistake and evaluate it leads
to overestimating the final RE performance by around 5% on ACE05. We also seize
this opportunity to study the unexplored ablations of two recent developments:
the use of language model pretraining (specifically BERT) and span-level NER.
This meta-analysis emphasizes the need for rigor in the report of both the
evaluation setting and the datasets statistics and we call for unifying the
evaluation setting in end-to-end RE.

本文针对目前关系提取领域中性能比较不可靠的问题，提出了一些无效比较的模式，并进行了实证研究来量化最常见错误对最终关系提取性能的影响，揭示了 BERT 和 span-level NER 的不足。本文强调在评估环节中应统一评估设置和数据集统计，并呼吁在 end-to-end 关系提取领域建立公认的评估标准。