Generated texts from large pretrained language models have been shown to
exhibit a variety of harmful, human-like biases about various demographics.
These findings prompted large efforts aiming to understand and measure such
effects, with the goal of providing benchmarks that can guide the development
of techniques mitigating these stereotypical associations. However, as recent
research has pointed out, the current benchmarks lack a robust experimental
setup, consequently hindering the inference of meaningful conclusions from
their evaluation metrics. In this paper, we extend these arguments and
demonstrate that existing techniques and benchmarks aiming to measure
stereotypes tend to be inaccurate and consist of a high degree of experimental
noise that severely limits the knowledge we can gain from benchmarking language
models based on them. Accordingly, we propose a new framework for robustly
measuring and quantifying biases exhibited by generative language models.
Finally, we use this framework to investigate GPT-3's occupational gender bias
and propose prompting techniques for mitigating these biases without the need
for fine-tuning.

本文研究了预训练语言模型的生成文本中存在的人类偏见和其对不同人口群体的影响，针对现有的评估技术和基准的准确性问题，提出了新的评估框架以更稳健地测量和量化语言模型所表现出的偏见。并使用该框架调查了 GPT-3 的职业性别偏见，并提出了一些缓解这些偏见的提示技术。