We study the robustness of machine reading comprehension (MRC) models to
entity renaming -- do models make more wrong predictions when the same
questions are asked about an entity whose name has been changed? Such failures
imply that models overly rely on entity information to answer questions, and
thus may generalize poorly when facts about the world change or questions are
asked about novel entities. To systematically audit this issue, we present a
pipeline to automatically generate test examples at scale, by replacing entity
names in the original test sample with names from a variety of sources, ranging
from names in the same test set, to common names in life, to arbitrary strings.
Across five datasets and three pretrained model architectures, MRC models
consistently perform worse when entities are renamed, with particularly large
accuracy drops on datasets constructed via distant supervision. We also find
large differences between models: SpanBERT, which is pretrained with span-level
masking, is more robust than RoBERTa, despite having similar accuracy on
unperturbed test data. We further experiment with different masking strategies
as the continual pretraining objective and find that entity-based masking can
improve the robustness of MRC models.

研究机器阅读理解模型对实体更名的鲁棒性，并发现实体更名对于 MRC 模型的预测准确率有显著影响。通过使用各种不同的源数据自动替换实体名称，我们在五个数据集和三个预训练模型结构上验证了该问题，并发现 SpanBERT 模型在实体更名下具有更好的鲁棒性。此外，我们还尝试了不同的遮蔽策略，发现基于实体遮蔽的方法能够提高 MRC 模型的鲁棒性。