Recent studies indicate that NLU models are prone to rely on shortcut
features for prediction, without achieving true language understanding. As a
result, these models fail to generalize to real-world out-of-distribution data.
In this work, we show that the words in the NLU training set can be modeled as
a long-tailed distribution. There are two findings: 1) NLU models have strong
preference for features located at the head of the long-tailed distribution,
and 2) Shortcut features are picked up during very early few iterations of the
model training. These two observations are further employed to formulate a
measurement which can quantify the shortcut degree of each training sample.
Based on this shortcut measurement, we propose a shortcut mitigation framework
LTGR, to suppress the model from making overconfident predictions for samples
with large shortcut degree. Experimental results on three NLU benchmarks
demonstrate that our long-tailed distribution explanation accurately reflects
the shortcut learning behavior of NLU models. Experimental analysis further
indicates that LTGR can improve the generalization accuracy on OOD data, while
preserving the accuracy on in-distribution data.

本文研究表明，NLU 模型倾向于依赖于快捷特征进行预测，而不是真正理解语言。我们发现 NLU 训练集中的单词可以被建模为长尾分布，进一步提出了一种量化每个训练样例快捷程度的方法，并基于此提出了一个缓解快捷特征对模型训练的影响的框架 LTGR，实验结果表明这种方法可以提高模型对 OOD 数据的泛化能力，同时保持在分布数据上的准确性。