Perceptions of hate can vary greatly across cultural contexts. Hate speech (HS) datasets, however, have traditionally been developed by language. This hides potential cultural biases, as one language may be spoken in different countries home to different cultures. In this work, we evaluate cultural bias in HS datasets by leveraging two interrelated cultural proxies: language and geography. We conduct a systematic survey of HS datasets in eight languages and confirm past findings on their English-language bias, but also show that this bias has been steadily decreasing in the past few years. For three geographically-widespread languages -- English, Arabic and Spanish -- we then leverage geographical metadata from tweets to approximate geo-cultural contexts by pairing language and country information. We find that HS datasets for these languages exhibit a strong geo-cultural bias, largely overrepresenting a handful of countries (e.g., US and UK for English) relative to their prominence in both the broader social media population and the general population speaking these languages. Based on these findings, we formulate recommendations for the creation of future HS datasets.

在本研究中，我们评估了仇恨言论数据集中的文化偏见，通过利用语言和地理两个互相关联的文化代理。我们对八种语言的仇恨言论数据集进行了系统调查，验证了对其英语偏见的先前研究结果，但也显示出这种偏见在过去几年中逐渐减少。对于英语、阿拉伯语和西班牙语这三种地理分布广泛的语言，我们利用推特的地理元数据来近似地理文化背景，通过将语言和国家信息配对。我们发现这些语言的仇恨言论数据集存在强烈的地理文化偏见，相对于在更广泛的社交媒体人口和讲这些语言的人口中的重要性，它们在很大程度上过度代表了少数几个国家（例如，对于英语来说是美国和英国）。基于这些结果，我们提出了创建未来仇恨言论数据集的建议。

从语言到地理：评估仇恨言论数据集中的文化偏见