As generative large language models (LLMs) grow more performant and
prevalent, we must develop comprehensive enough tools to measure and improve
their fairness. Different prompt-based datasets can be used to measure social
bias across multiple text domains and demographic axes, meaning that testing
LLMs on more datasets can potentially help us characterize their biases more
fully, and better ensure equal and equitable treatment of marginalized
demographic groups. In this work, our focus is two-fold:
(1) Benchmarking: a comparison of 6 different prompt-based bias and toxicity
metrics across 12 demographic axes and 5 families of generative LLMs. Out of
those 6 metrics, AdvPromptSet and HolisticBiasR are novel datasets proposed in
the paper. The comparison of those benchmarks gives us insights about the bias
and toxicity of the compared models. Therefore, we explore the frequency of
demographic terms in common LLM pre-training corpora and how this may relate to
model biases.
(2) Mitigation: we conduct a comprehensive study of how well 3 bias/toxicity
mitigation techniques perform across our suite of measurements. ROBBIE aims to
provide insights for practitioners while deploying a model, emphasizing the
need to not only measure potential harms, but also understand how they arise by
characterizing the data, mitigate harms once found, and balance any trade-offs.
We open-source our analysis code in hopes of encouraging broader measurements
of bias in future LLMs.

评估和改善大型语言模型（LLMs）的公平性，通过使用不同的提示性数据集来测量社会偏见，对比模型之间的偏差和毒性度量，并研究偏差 / 毒性降低技术的效果。我们开源分析代码以鼓励测量未来 LLMs 偏见的广泛研究。