Recent research has shown that large language models rely on spurious
correlations in the data for natural language understanding (NLU) tasks. In
this work, we aim to answer the following research question: Can we reduce
spurious correlations by modifying the ground truth labels of the training
data? Specifically, we propose a simple yet effective debiasing framework,
named Soft Label Encoding (SoftLE). We first train a teacher model with hard
labels to determine each sample's degree of relying on shortcuts. We then add
one dummy class to encode the shortcut degree, which is used to smooth other
dimensions in the ground truth label to generate soft labels. This new ground
truth label is used to train a more robust student model. Extensive experiments
on two NLU benchmark tasks demonstrate that SoftLE significantly improves
out-of-distribution generalization while maintaining satisfactory
in-distribution accuracy.

最近的研究表明，大型语言模型在自然语言理解任务中依赖于数据中的伪相关性。本研究旨在回答以下研究问题：我们能否通过改变训练数据的真实标签来减少伪相关性？具体而言，我们提出了一种简单而有效的去偏框架，称为软标签编码（SoftLE）。我们首先使用硬标签训练一个教师模型来确定每个样本对快捷方式的依赖程度。然后，我们添加一个虚拟类别来编码快捷方式程度，该虚拟类别用于平滑地生成其他维度上的地面真实标签生成软标签。这个新的地面真实标签用于训练一个更强大的学生模型。对两个自然语言理解基准任务进行的广泛实验证明，SoftLE 在保持令人满意的内部准确性的同时，显著提高了超过分布的概括能力。