Large-scale pre-trained language models have demonstrated high performance on standard datasets for natural language inference (NLI) tasks. Unfortunately, these evaluations can be misleading, as although the models can perform well on in-distribution data, they perform poorly on out-of-distribution test sets, such as contrast sets. Contrast sets consist of perturbed instances of data that have very minor, but meaningful, changes to the input that alter the gold label, revealing how models can learn superficial patterns in the training data rather than learning more sophisticated language nuances. As an example, the ELECTRA-small language model achieves nearly 90% accuracy on an SNLI dataset but drops to 75% when tested on an out-of-distribution contrast set. The research carried out in this study explores how the robustness of a language model can be improved by exposing it to small amounts of more complex contrast sets during training to help it better learn language patterns. With this approach, the model recovers performance and achieves nearly 90% accuracy on contrast sets, highlighting the importance of diverse and challenging training data.

本研究针对语言模型在对比集上的表现不佳这一问题，提出通过在训练过程中引入更多复杂对比集的方式来提升模型的鲁棒性。研究发现，这种方法能显著提升模型对语言模式的理解能力，使其在对比集上的准确率达到近90%，强调了多样化和具有挑战性的训练数据的重要性。

从表面模式到语义理解：对比集上微调语言模型