Many datasets have been shown to contain incidental correlations created by
idiosyncrasies in the data collection process. For example, sentence entailment
datasets can have spurious word-class correlations if nearly all contradiction
sentences contain the word "not", and image recognition datasets can have
tell-tale object-background correlations if dogs are always indoors. In this
paper, we propose a method that can automatically detect and ignore these kinds
of dataset-specific patterns, which we call dataset biases. Our method trains a
lower capacity model in an ensemble with a higher capacity model. During
training, the lower capacity model learns to capture relatively shallow
correlations, which we hypothesize are likely to reflect dataset bias. This
frees the higher capacity model to focus on patterns that should generalize
better. We ensure the models learn non-overlapping approaches by introducing a
novel method to make them conditionally independent. Importantly, our approach
does not require the bias to be known in advance. We evaluate performance on
synthetic datasets, and four datasets built to penalize models that exploit
known biases on textual entailment, visual question answering, and image
recognition tasks. We show improvement in all settings, including a 10 point
gain on the visual question answering dataset.

本文提出了一种检测和忽略数据集特异性模式的方法，采用高低容量模型的集成训练，以自动捕捉相对浅层的相关性，并确保两个模型学习不重叠的方法，以利于更好地泛化模式。