Large Language Models (LLMs) can produce responses that exhibit social biases
and support stereotypes. However, conventional benchmarking is insufficient to
thoroughly evaluate LLM bias, as it can not scale to large sets of prompts and
provides no guarantees. Therefore, we propose a novel certification framework
QuaCer-B (Quantitative Certification of Bias) that provides formal guarantees
on obtaining unbiased responses from target LLMs under large sets of prompts. A
certificate consists of high-confidence bounds on the probability of obtaining
biased responses from the LLM for any set of prompts containing sensitive
attributes, sampled from a distribution. We illustrate the bias certification
in LLMs for prompts with various prefixes drawn from given distributions. We
consider distributions of random token sequences, mixtures of manual
jailbreaks, and jailbreaks in the LLM's embedding space to certify its bias. We
certify popular LLMs with QuaCer-B and present novel insights into their
biases.

大型语言模型（LLMs）可以产生展现社交偏见并支持刻板印象的回答。然而，传统的基准测试无法充分评估 LLM 偏见，因为它无法扩展到大量的提示集，并且没有提供保证。因此，我们提出了一种新的认证框架 QuaCer-B（Bias 的定量认证），它提供了在大量提示集下从目标 LLMs 获取无偏回答的形式保证。证书包括从分布中采样的包含敏感属性的任何提示集获得有偏回答的概率的高置信度上限。我们通过在给定分布中随机令牌序列、手动越狱的混合和 LLM 嵌入空间中的越狱的提示上对 LLM 的偏见进行证明。我们使用 QuaCer-B 对流行的 LLMs 进行认证，并呈现关于其偏见的新见解。