Models for natural language understanding (NLU) tasks often rely on the
idiosyncratic biases of the dataset, which make them brittle against test cases
outside the training distribution. Recently, several proposed debiasing methods
are shown to be very effective in improving out-of-distribution performance.
However, their improvements come at the expense of performance drop when models
are evaluated on the in-distribution data, which contain examples with higher
diversity. This seemingly inevitable trade-off may not tell us much about the
changes in the reasoning and understanding capabilities of the resulting models
on broader types of examples beyond the small subset represented in the
out-of-distribution data. In this paper, we address this trade-off by
introducing a novel debiasing method, called confidence regularization, which
discourage models from exploiting biases while enabling them to receive enough
incentive to learn from all the training examples. We evaluate our method on
three NLU tasks and show that, in contrast to its predecessors, it improves the
performance on out-of-distribution datasets (e.g., 7pp gain on HANS dataset)
while maintaining the original in-distribution accuracy.

本文介绍了一种名为置信度正则化的新型去偏方法，旨在提高自然语言理解任务模型对于越界数据的表现，同时保持对内界数据的准确度。实验表明，与之前的方法相比，该方法在提高模型对越界数据的表现方面具有更好的效果。

注重权衡：在不损害内部分布性能的情况下去偏向自然语言理解模型

Mind the Trade-off: Debiasing NLU Models without Degrading the  In-distribution Performance

Recent advances in domain adaptation show that deep self-training presents a
powerful means for unsupervised domain adaptation. These methods often involve
an iterative process of predicting on target domain and then taking the
confident predictions as pseudo-labels for retraining. However, since
pseudo-labels can be noisy, self-training can put overconfident label belief on
wrong classes, leading to deviated solutions with propagated errors. To address
the problem, we propose a confidence regularized self-training (CRST)
framework, formulated as regularized self-training. Our method treats
pseudo-labels as continuous latent variables jointly optimized via alternating
optimization. We propose two types of confidence regularization: label
regularization (LR) and model regularization (MR). CRST-LR generates soft
pseudo-labels while CRST-MR encourages the smoothness on network output.
Extensive experiments on image classification and semantic segmentation show
that CRSTs outperform their non-regularized counterpart with state-of-the-art
performance. The code and models of this work are available at
this https URL

本文提出了一种基于置信度正则化的自训练框架，在目标域上进行伪标签预测，并通过交替优化联合优化伪标签来实现领域自适应，其中伪标签作为连续隐变量进行处理，该方法在图像分类和语义分割中实验证明了其具有优异的性能。