The CoNLL-03 corpus is arguably the most well-known and utilized benchmark
dataset for named entity recognition (NER). However, prior works found
significant numbers of annotation errors, incompleteness, and inconsistencies
in the data. This poses challenges to objectively comparing NER approaches and
analyzing their errors, as current state-of-the-art models achieve F1-scores
that are comparable to or even exceed the estimated noise level in CoNLL-03. To
address this issue, we present a comprehensive relabeling effort assisted by
automatic consistency checking that corrects 7.0% of all labels in the English
CoNLL-03. Our effort adds a layer of entity linking annotation both for better
explainability of NER labels and as additional safeguard of annotation quality.
Our experimental evaluation finds not only that state-of-the-art approaches
reach significantly higher F1-scores (97.1%) on our data, but crucially that
the share of correct predictions falsely counted as errors due to annotation
noise drops from 47% to 6%. This indicates that our resource is well suited to
analyze the remaining errors made by state-of-the-art models, and that the
theoretical upper bound even on high resource, coarse-grained NER is not yet
reached. To facilitate such analysis, we make CleanCoNLL publicly available to
the research community.

通过综合重新标定工作和自动一致性检查，修正了英文 CoNLL-03 中所有标签的 7.0%。我们的资源通过实验证实了最先进的方法在我们的数据上达到了显著更高的 F1 得分（97.1%），并且通过注释噪声的误差分析发现高资源粗粒度 NER 的理论上限尚未达到。