Modern named entity recognition systems have steadily improved performance in
the age of larger and more powerful neural models. However, over the past
several years, the state-of-the-art has seemingly hit another plateau on the
benchmark CoNLL-03 English dataset. In this paper, we perform a deep dive into
the test outputs of the highest-performing NER models, conducting a
fine-grained evaluation of their performance by introducing new document-level
annotations on the test set. We go beyond F1 scores by categorizing errors in
order to interpret the true state of the art for NER and guide future work. We
review previous attempts at correcting the various flaws of the test set and
introduce CoNLL#, a new corrected version of the test set that addresses its
systematic and most prevalent errors, allowing for low-noise, interpretable
error analysis.

现代命名实体识别系统在更大更强大的神经模型时代中稳步提高性能。本文通过对最高性能 NER 模型的测试输出进行深入分析，并在测试集上引入新的文档级注释，对其性能进行精细评估。我们通过对错误进行分类，超越 F1 分数，解释 NER 的真实技术水平并指导未来的研究。我们回顾了之前纠正测试集各种缺陷的尝试，并引入了一个新的纠正版本 CoNLL#，解决了其系统性和最常见的错误，从而允许进行低噪声、可解释的错误分析。