Language Models (LMs) have been shown to exhibit a strong preference towards entities associated with Western culture when operating in non-Western languages. In this paper, we aim to uncover the origins of entity-related cultural biases in LMs by analyzing several contributing factors, including the representation of entities in pre-training data and the impact of variations in linguistic phenomena across languages. We introduce CAMeL-2, a parallel Arabic-English benchmark of 58,086 entities associated with Arab and Western cultures and 367 masked natural contexts for entities. Our evaluations using CAMeL-2 reveal reduced performance gaps between cultures by LMs when tested in English compared to Arabic. We find that LMs struggle in Arabic with entities that appear at high frequencies in pre-training, where entities can hold multiple word senses. This also extends to entities that exhibit high lexical overlap with languages that are not Arabic but use the Arabic script. Further, we show how frequency-based tokenization leads to this issue in LMs, which gets worse with larger Arabic vocabularies. We will make CAMeL-2 available at: https://github.com/tareknaous/camel2

本研究探讨了语言模型在非西方语言中对西方文化实体的偏好，揭示了实体相关文化偏见的来源。通过引入CAMeL-2基准，评估了阿拉伯语和英语中的实体表现，发现语言模型在阿拉伯语中的表现差异以及频率基础的标记化导致的文化偏见加剧。此工作的潜在影响在于优化多语言模型的公平性与准确性。

语言模型中文化偏见的起源：从预训练数据到语言现象