Fine-tuning pretrained model has achieved promising performance on standard
NER benchmarks. Generally, these benchmarks are blessed with strong name
regularity, high mention coverage and sufficient context diversity.
Unfortunately, when scaling NER to open situations, these advantages may no
longer exist. And therefore it raises a critical question of whether previous
creditable approaches can still work well when facing these challenges. As
there is no currently available dataset to investigate this problem, this paper
proposes to conduct randomization test on standard benchmarks. Specifically, we
erase name regularity, mention coverage and context diversity respectively from
the benchmarks, in order to explore their impact on the generalization ability
of models. To further verify our conclusions, we also construct a new open NER
dataset that focuses on entity types with weaker name regularity and lower
mention coverage to verify our conclusion. From both randomization test and
empirical experiments, we draw the conclusions that 1) name regularity is
critical for the models to generalize to unseen mentions; 2) high mention
coverage may undermine the model generalization ability and 3) context patterns
may not require enormous data to capture when using pretrained encoders.

本文研究在 fine-tuning 预训练模型的情况下，在命名实体识别任务中如何面对开放环境中可能面临的挑战，作者进行了随机测试和实证实验，结果表明，命名规则对于模型推广到未见过的提及至关重要，高覆盖率可能削弱模型的推广能力，而上下文模式则不需要使用大量数据来捕捉。