A growing body of work shows that models exploit annotation artifacts to
achieve state-of-the-art performance on standard crowdsourced
benchmarks---datasets collected from crowdworkers to create an evaluation
task---while still failing on out-of-domain examples for the same task. Recent
work has explored the use of counterfactually-augmented data---data built by
minimally editing a set of seed examples to yield counterfactual labels---to
augment training data associated with these benchmarks and build more robust
classifiers that generalize better. However, Khashabi et al. (2020) find that
this type of augmentation yields little benefit on reading comprehension tasks
when controlling for dataset size and cost of collection. We build upon this
work by using English natural language inference data to test model
generalization and robustness and find that models trained on a
counterfactually-augmented SNLI dataset do not generalize better than
unaugmented datasets of similar size and that counterfactual augmentation can
hurt performance, yielding models that are less robust to challenge examples.
Counterfactual augmentation of natural language understanding data through
standard crowdsourcing techniques does not appear to be an effective way of
collecting training data and further innovation is required to make this
general line of work viable.

本研究发现，通过常规众包技术建立的反事实数据增强自然语言阅读理解数据集，不利于训练模型的泛化性和鲁棒性，并且在成本考虑上使用此种方法带来的收益很少。