Named entity recognition systems perform well on standard datasets comprising English news. But given the paucity of data, it is difficult to draw conclusions about the robustness of systems with respect to recognizing a diverse set of entities. We propose a method for auditing the in-domain robustness of systems, focusing specifically on differences in performance due to the national origin of entities. We create entity-switched datasets, in which named entities in the original texts are replaced by plausible named entities of the same type but of different national origin. We find that state-of-the-art systems' performance vary widely even in-domain: In the same context, entities from certain origins are more reliably recognized than entities from elsewhere. Systems perform best on American and Indian entities, and worst on Vietnamese and Indonesian entities. This auditing approach can facilitate the development of more robust named entity recognition systems, and will allow research in this area to consider fairness criteria that have received heightened attention in other predictive technology work.

本文提出了一种审计命名实体识别系统的方法，在考虑实体国籍因素的前提下，通过创建实体交换数据集来分析系统的域内稳健性，并发现系统的性能表现存在显著差异，其中美国和印度实体的识别效果最佳，而越南和印尼实体的识别效果最差，这种审计方法有利于发展更具稳健性的命名实体识别系统，同时为研究公平性标准提供了参考。

实体交换数据集: 一种审计命名实体识别模型领域内鲁棒性的方法