Named Entity Recognition (NER) is a well researched NLP task and is widely
used in real world NLP scenarios. NER research typically focuses on the
creation of new ways of training NER, with relatively less emphasis on
resources and evaluation. Further, state of the art (SOTA) NER models, trained
on standard datasets, typically report only a single performance measure
(F-score) and we don't really know how well they do for different entity types
and genres of text, or how robust are they to new, unseen entities. In this
paper, we perform a broad evaluation of NER using a popular dataset, that takes
into consideration various text genres and sources constituting the dataset at
hand. Additionally, we generate six new adversarial test sets through small
perturbations in the original test set, replacing select entities while
retaining the context. We also train and test our models on randomly generated
train/dev/test splits followed by an experiment where the models are trained on
a select set of genres but tested genres not seen in training. These
comprehensive evaluation strategies were performed using three SOTA NER models.
Based on our results, we recommend some useful reporting practices for NER
researchers, that could help in providing a better understanding of a SOTA
model's performance in future.

本文通过对多种文本来源进行广泛评估以及通过构建新的对抗性测试集合，试图评估目前三种最先进的命名实体识别模型。在评估中，我们呈现了一些有用的报告实践方法，以便于提高当前模型性能的理解和提高未来研究的效率。

关于现有顶尖 NER 技术状态我们真正了解了什么？

What do we Really Know about State of the Art NER?

Recent techniques in Question Answering (QA) have gained remarkable
performance improvement with some QA models even surpassed human performance.
However, the ability of these models in truly understanding the language still
remains dubious and the models are revealing limitations when facing
adversarial examples. To strengthen the robustness of QA models and their
generalization ability, we propose a representation Enhancement via Semantic
and Context constraints (ESC) approach to improve the robustness of lexical
embeddings. Specifically, we insert perturbations with semantic constraints and
train enhanced contextual representations via a context-constraint loss to
better distinguish the context clues for the correct answer. Experimental
results show that our approach gains significant robustness improvement on four
adversarial test sets.

提出一种基于语义和上下文限制的表示增强方法，来改善问题回答模型的强壮性和泛化能力，通过对语义进行扰动和训练增强上下文表示来更好地区分正确答案的上下文线索，实验证明我们的方法在四个对抗性测试集上获得了显著的鲁棒性改善。