Question answering (QA) models have shown compelling results in the task of
Machine Reading Comprehension (MRC). Recently these systems have proved to
perform better than humans on held-out test sets of datasets e.g. SQuAD, but
their robustness is not guaranteed. The QA model's brittleness is exposed when
evaluated on adversarial generated examples by a performance drop. In this
study, we explore the robustness of MRC models to entity renaming, with
entities from low-resource regions such as Africa. We propose EntSwap, a method
for test-time perturbations, to create a test set whose entities have been
renamed. In particular, we rename entities of type: country, person,
nationality, location, organization, and city, to create AfriSQuAD2. Using the
perturbed test set, we evaluate the robustness of three popular MRC models. We
find that compared to base models, large models perform well comparatively on
novel entities. Furthermore, our analysis indicates that entity type person
highly challenges the MRC models' performance.

本研究旨在探究机器阅读理解模型对命名实体重命名的鲁棒性，尤其是对低资源地区如非洲的实体。通过创建一个被重命名的数据集 AfriSQuAD2 来测试模型的鲁棒性，并发现大模型相对于基础模型表现更好，而人类实体类型则高度挑战 MRC 模型的性能。

评估机器阅读理解模型对低资源实体重命名的鲁棒性

Evaluating the Robustness of Machine Reading Comprehension Models to Low  Resource Entity Renaming

We study the robustness of machine reading comprehension (MRC) models to
entity renaming -- do models make more wrong predictions when the same
questions are asked about an entity whose name has been changed? Such failures
imply that models overly rely on entity information to answer questions, and
thus may generalize poorly when facts about the world change or questions are
asked about novel entities. To systematically audit this issue, we present a
pipeline to automatically generate test examples at scale, by replacing entity
names in the original test sample with names from a variety of sources, ranging
from names in the same test set, to common names in life, to arbitrary strings.
Across five datasets and three pretrained model architectures, MRC models
consistently perform worse when entities are renamed, with particularly large
accuracy drops on datasets constructed via distant supervision. We also find
large differences between models: SpanBERT, which is pretrained with span-level
masking, is more robust than RoBERTa, despite having similar accuracy on
unperturbed test data. We further experiment with different masking strategies
as the continual pretraining objective and find that entity-based masking can
improve the robustness of MRC models.

研究机器阅读理解模型对实体更名的鲁棒性，并发现实体更名对于 MRC 模型的预测准确率有显著影响。通过使用各种不同的源数据自动替换实体名称，我们在五个数据集和三个预训练模型结构上验证了该问题，并发现 SpanBERT 模型在实体更名下具有更好的鲁棒性。此外，我们还尝试了不同的遮蔽策略，发现基于实体遮蔽的方法能够提高 MRC 模型的鲁棒性。