Several datasets have recently been constructed to expose brittleness in
models trained on existing benchmarks. While model performance on these
challenge datasets is significantly lower compared to the original benchmark,
it is unclear what particular weaknesses they reveal. For example, a challenge
dataset may be difficult because it targets phenomena that current models
cannot capture, or because it simply exploits blind spots in a model's specific
training set. We introduce inoculation by fine-tuning, a new analysis method
for studying challenge datasets by exposing models (the metaphorical patient)
to a small amount of data from the challenge dataset (a metaphorical pathogen)
and assessing how well they can adapt. We apply our method to analyze the NLI
"stress tests" (Naik et al., 2018) and the Adversarial SQuAD dataset (Jia and
Liang, 2017). We show that after slight exposure, some of these datasets are no
longer challenging, while others remain difficult. Our results indicate that
failures on challenge datasets may lead to very different conclusions about
models, training datasets, and the challenge datasets themselves.

本文介绍了一种新的分析方法，即通过微调接种的方法研究挑战数据集，应用于 NLI “压力测试” 和 Adversarial SQuAD 数据集，结果表明，挑战数据集的失败可能会导致关于模型，训练数据集和挑战数据集本身的完全不同的结论。