In the domain of Natural Language Inference (NLI), especially in tasks
involving the classification of multiple input texts, the Cross-Entropy Loss
metric is widely employed as a standard for error measurement. However, this
metric falls short in effectively evaluating a model's capacity to understand
language entailments. In this study, we introduce an innovative technique for
generating a contrast set for the Stanford Natural Language Inference (SNLI)
dataset. Our strategy involves the automated substitution of verbs, adverbs,
and adjectives with their synonyms to preserve the original meaning of
sentences. This method aims to assess whether a model's performance is based on
genuine language comprehension or simply on pattern recognition. We conducted
our analysis using the ELECTRA-small model. The model achieved an accuracy of
89.9% on the conventional SNLI dataset but showed a reduced accuracy of 72.5%
on our contrast set, indicating a substantial 17% decline. This outcome led us
to conduct a detailed examination of the model's learning behaviors. Following
this, we improved the model's resilience by fine-tuning it with a
contrast-enhanced training dataset specifically designed for SNLI, which
increased its accuracy to 85.5% on the contrast sets. Our findings highlight
the importance of incorporating diverse linguistic expressions into datasets
for NLI tasks. We hope that our research will encourage the creation of more
inclusive datasets, thereby contributing to the development of NLI models that
are both more sophisticated and effective.

在自然语言推理领域中，交叉熵损失度量被广泛应用于多输入文本分类任务，但该度量方式在有效评估模型理解语义蕴涵能力方面存在不足。本研究引入一种创新技术，通过自动替换动词、副词和形容词的近义词来产生斯坦福自然语言推理数据集的对比集，以评估模型表现是否基于真实语言理解还是仅仅基于模式识别。研究使用 ELECTRA-small 模型进行分析，模型在常规的自然语言推理数据集上达到 89.9% 的准确率，但在对比集上准确率降低到了 72.5%，下降了 17%。由此结果导致我们对模型的学习行为进行了详细考察，然后通过针对自然语言推理数据集的对比增强训练数据集对模型进行微调，将其准确率提升到了 85.5%。我们的研究结果强调了在自然语言推理任务中加入多样化的语言表达的重要性，希望能促进创建更包容的数据集，从而推动更为复杂和有效的自然语言推理模型的发展。