Gender-bias stereotypes have recently raised significant ethical concerns in natural language processing. However, progress in detection and evaluation of gender bias in natural language understanding through inference is limited and requires further investigation. In this work, we propose an evaluation methodology to measure these biases by constructing a challenge task that involves pairing gender-neutral premises against a gender-specific hypothesis. We use our challenge task to investigate state-of-the-art NLI models on the presence of gender stereotypes using occupations. Our findings suggest that three models (BERT, RoBERTa, BART) trained on MNLI and SNLI datasets are significantly prone to gender-induced prediction errors. We also find that debiasing techniques such as augmenting the training dataset to ensure a gender-balanced dataset can help reduce such bias in certain cases.

本文提出了一种评估方法来测量天生的性别偏见，其通过构建一项挑战任务，通过将性别中性前提与性别具体假设进行配对的方式来进行。研究发现，许多先进的NLI模型在使用职业数据集进行训练时会因性别偏见而出现错误，但通过通过扩充训练数据集来确保性别平衡，可以在某些情况下帮助减少此类偏差。

自然语言推断中的性别偏见评估