Neural networks trained with ERM (empirical risk minimization) sometimes
learn unintended decision rules, in particular when their training data is
biased, i.e., when training labels are strongly correlated with undesirable
features. To prevent a network from learning such features, recent methods
augment training data such that examples displaying spurious correlations
(i.e., bias-aligned examples) become a minority, whereas the other,
bias-conflicting examples become prevalent. However, these approaches are
sometimes difficult to train and scale to real-world data because they rely on
generative models or disentangled representations. We propose an alternative
based on mixup, a popular augmentation that creates convex combinations of
training examples. Our method, coined SelecMix, applies mixup to contradicting
pairs of examples, defined as showing either (i) the same label but dissimilar
biased features, or (ii) different labels but similar biased features.
Identifying such pairs requires comparing examples with respect to unknown
biased features. For this, we utilize an auxiliary contrastive model with the
popular heuristic that biased features are learned preferentially during
training. Experiments on standard benchmarks demonstrate the effectiveness of
the method, in particular when label noise complicates the identification of
bias-conflicting examples.

提出一种新的数据增广方法 SelecMix，旨在解决神经网络训练中数据偏差带来的问题，并在标准基准测试中证明了其有效性，尤其是在标签噪声情况下。

SelecMix: 通过对立对采样进行偏差学习

SelecMix: Debiased Learning by Contradicting-pair Sampling

Fact verification requires validating a claim in the context of evidence. We
show, however, that in the popular FEVER dataset this might not necessarily be
the case. Claim-only classifiers perform competitively with top evidence-aware
models. In this paper, we investigate the cause of this phenomenon, identifying
strong cues for predicting labels solely based on the claim, without
considering any evidence. We create an evaluation set that avoids those
idiosyncrasies. The performance of FEVER-trained models significantly drops
when evaluated on this test set. Therefore, we introduce a regularization
method which alleviates the effect of bias in the training data, obtaining
improvements on the newly created test set. This work is a step towards a more
sound evaluation of reasoning capabilities in fact verification models.

在研究中，我们发现在常见的 FEVER 数据集中，仅针对声明的分类器与最佳证据感知模型具有相似的性能。为了解决这种现象，我们开发了一个评估集并引入了一种正则化方法以减少训练数据中偏差的影响，并取得了进展。该工作是朝着更加可靠地评估事实验证模型的推理能力迈出的一步。