Neural Machine Translation (NMT) models, though state-of-the-art for
translation, often reflect social biases, particularly gender bias. Existing
evaluation benchmarks primarily focus on English as the source language of
translation. For source languages other than English, studies often employ
gender-neutral sentences for bias evaluation, whereas real-world sentences
frequently contain gender information in different forms. Therefore, it makes
more sense to evaluate for bias using such source sentences to determine if NMT
models can discern gender from the grammatical gender cues rather than relying
on biased associations. To illustrate this, we create two gender-specific
sentence sets in Hindi to automatically evaluate gender bias in various
Hindi-English (HI-EN) NMT systems. We emphasise the significance of tailoring
bias evaluation test sets to account for grammatical gender markers in the
source language.

神经机器翻译中存在性别偏见，而评估基准主要关注英语作为源语言的翻译，我们提出了为源语言中的语法性别标记量身定制偏见评估测试集的方法，以确定 NMT 模型是否能通过语法性别线索来区分性别。