Implicit gender bias in Large Language Models (LLMs) is a well-documented
problem, and implications of gender introduced into automatic translations can
perpetuate real-world biases. However, some LLMs use heuristics or
post-processing to mask such bias, making investigation difficult. Here, we
examine bias in LLMss via back-translation, using the DeepL translation API to
investigate the bias evinced when repeatedly translating a set of 56 Software
Engineering tasks used in a previous study. Each statement starts with 'she',
and is translated first into a 'genderless' intermediate language then back
into English; we then examine pronoun- choice in the back-translated texts. We
expand prior research in the following ways: (1) by comparing results across
five intermediate languages, namely Finnish, Indonesian, Estonian, Turkish and
Hungarian; (2) by proposing a novel metric for assessing the variation in
gender implied in the repeated translations, avoiding the over-interpretation
of individual pronouns, apparent in earlier work; (3) by investigating sentence
features that drive bias; (4) and by comparing results from three time-lapsed
datasets to establish the reproducibility of the approach. We found that some
languages display similar patterns of pronoun use, falling into three loose
groups, but that patterns vary between groups; this underlines the need to work
with multiple languages. We also identify the main verb appearing in a sentence
as a likely significant driver of implied gender in the translations. Moreover,
we see a good level of replicability in the results, and establish that our
variation metric proves robust despite an obvious change in the behaviour of
the DeepL translation API during the course of the study. These results show
that the back-translation method can provide further insights into bias in
language models.

通过反向翻译方法研究大型语言模型中的内隐性别偏见，比较不同语言的结果，提出一种新的评估性别隐含变化的度量标准，并探究驱动偏见的句子特征，结果表明该方法能够进一步揭示语言模型中的偏见。

调查机器翻译中性别偏见的标记和驱动因素

Investigating Markers and Drivers of Gender Bias in Machine Translations

Large Language Models (LLMs) can generate biased responses. Yet previous
direct probing techniques contain either gender mentions or predefined gender
stereotypes, which are challenging to comprehensively collect. Hence, we
propose an indirect probing framework based on conditional generation. This
approach aims to induce LLMs to disclose their gender bias even without
explicit gender or stereotype mentions. We explore three distinct strategies to
disclose explicit and implicit gender bias in LLMs. Our experiments demonstrate
that all tested LLMs exhibit explicit and/or implicit gender bias, even when
gender stereotypes are not present in the inputs. In addition, an increased
model size or model alignment amplifies bias in most cases. Furthermore, we
investigate three methods to mitigate bias in LLMs via Hyperparameter Tuning,
Instruction Guiding, and Debias Tuning. Remarkably, these methods prove
effective even in the absence of explicit genders or stereotypes.

大语言模型可以生成具有偏见的回答。然而，先前的直接探测技术包含性别提及或预定义的性别刻板印象，这些很难全面收集。因此，我们提出了一种基于条件生成的间接探测框架，旨在诱使大语言模型揭示其性别偏见，即使没有显式的性别或刻板印象提及。我们探索了三种不同的策略来揭示大语言模型中的显式和隐式性别偏见。我们的实验证明，所有经过测试的大语言模型都表现出显式和 / 或隐式的性别偏见，即使输入中没有性别刻板印象。此外，模型大小的增加或模型对齐会放大大多数情况下的偏见。此外，我们通过超参数调整、指导性指导和去偏调整这三种方法来研究大语言模型中的偏见缓解。值得注意的是，即使没有显式的性别或刻板印象，这些方法也被证明是有效的。

LLMs 中性别偏见的揭示与减轻

Disclosure and Mitigation of Gender Bias in LLMs

Implicit gender bias in software development is a well-documented issue, such
as the association of technical roles with men. To address this bias, it is
important to understand it in more detail. This study uses data mining
techniques to investigate the extent to which 56 tasks related to software
development, such as assigning GitHub issues and testing, are affected by
implicit gender bias embedded in large language models. We systematically
translated each task from English into a genderless language and back, and
investigated the pronouns associated with each task. Based on translating each
task 100 times in different permutations, we identify a significant disparity
in the gendered pronoun associations with different tasks. Specifically,
requirements elicitation was associated with the pronoun "he" in only 6% of
cases, while testing was associated with "he" in 100% of cases. Additionally,
tasks related to helping others had a 91% association with "he" while the same
association for tasks related to asking coworkers was only 52%. These findings
reveal a clear pattern of gender bias related to software development tasks and
have important implications for addressing this issue both in the training of
large language models and in broader society.

本研究使用数据挖掘技术，调查了与软件开发相关的 56 项任务，发现不同任务中含有性别代词的比例差异显著，其中测试与帮助他人的任务中含有 “他” 这一性别代词比例显著较高，这些发现揭示了与软件开发任务相关的性别偏见模式。