Neural Machine Translation (NMT) models are state-of-the-art for machine
translation. However, these models are known to have various social biases,
especially gender bias. Most of the work on evaluating gender bias in NMT has
focused primarily on English as the source language. For source languages
different from English, most of the studies use gender-neutral sentences to
evaluate gender bias. However, practically, many sentences that we encounter do
have gender information. Therefore, it makes more sense to evaluate for bias
using such sentences. This allows us to determine if NMT models can identify
the correct gender based on the grammatical gender cues in the source sentence
rather than relying on biased correlations with, say, occupation terms. To
demonstrate our point, in this work, we use Hindi as the source language and
construct two sets of gender-specific sentences: OTSC-Hindi and WinoMT-Hindi
that we use to evaluate different Hindi-English (HI-EN) NMT systems
automatically for gender bias. Our work highlights the importance of
considering the nature of language when designing such extrinsic bias
evaluation datasets.

神经机器翻译（NMT）模型是机器翻译的先进技术，但这些模型被发现存在各种社会偏见，尤其是性别偏见。本文以印地语作为源语言，构建了两组性别特定的句子集（OTSC-Hindi 和 WinoMT-Hindi），用于自动评估不同的印地 - 英文（HI-EN）NMT 系统是否存在性别偏见。本研究强调了设计此类外在偏见评估数据集时考虑语言特性的重要性。