Masked Language Models (MLMs) have been successful in many natural language
processing tasks. However, real-world stereotype biases are likely to be
reflected in MLMs due to their learning from large text corpora. Most of the
evaluation metrics proposed in the past adopt different masking strategies,
designed with the log-likelihood of MLMs. They lack holistic considerations
such as variance for stereotype bias and anti-stereotype bias samples. In this
paper, the log-likelihoods of stereotype bias and anti-stereotype bias samples
output by MLMs are considered Gaussian distributions. Two evaluation metrics,
Kullback Leibler Divergence Score (KLDivS) and Jensen Shannon Divergence Score
(JSDivS) are proposed to evaluate social biases in MLMs The experimental
results on the public datasets StereoSet and CrowS-Pairs demonstrate that
KLDivS and JSDivS are more stable and interpretable compared to the metrics
proposed in the past.

本文利用高斯分布提出了 KLDivS 和 JSDivS 两种新的评估社会偏见的度量方法，并在公共数据集 StereoSet 和 CrowS-Pairs 上进行了实验，结果表明 KLDivS 和 JSDivS 比过去提出的度量方法更稳定和可解释。