Fairness has become a trending topic in natural language processing (NLP), which addresses biases targeting certain social groups such as genders and religions. However, regional bias in language models (LMs), a long-standing global discrimination problem, still remains unexplored. This paper bridges the gap by analysing the regional bias learned by the pre-trained language models that are broadly used in NLP tasks. In addition to verifying the existence of regional bias in LMs, we find that the biases on regional groups can be strongly influenced by the geographical clustering of the groups. We accordingly propose a HiErarchical Regional Bias evaluation method (HERB) utilising the information from the sub-region clusters to quantify the bias in pre-trained LMs. Experiments show that our hierarchical metric can effectively evaluate the regional bias with respect to comprehensive topics and measure the potential regional bias that can be propagated to downstream tasks. Our codes are available at https://github.com/Bernard-Yang/HERB.

本文针对自然语言处理中存在的公平性问题，提出了一种基于 HERB 方法的区域偏差评估方法，通过对 LMs 学习到的区域偏差进行分析，验证了其存在性，并发现集团地理聚集性对 LMs 的区域偏差具有显著影响。实验结果表明，我们的 hierarchical metric 方法可以有效评估 LMs 的区域偏差，并测量可能传播到下游任务的潜在区域偏差。

HERB：衡量预训练语言模型中的层次区域偏差