To tackle the global challenge of online hate speech, a large body of research has developed detection models to flag hate speech in the sea of online content. Yet, due to systematic biases in evaluation datasets, detection performance in real-world settings remains unclear, let alone across geographies. To address this issue, we introduce HateDay, the first global hate speech dataset representative of social media settings, randomly sampled from all tweets posted on September 21, 2022 for eight languages and four English-speaking countries. Using HateDay, we show how the prevalence and composition of hate speech varies across languages and countries. We also find that evaluation on academic hate speech datasets overestimates real-world detection performance, which we find is very low, especially for non-European languages. We identify several factors explaining poor performance, including models' inability to distinguish between hate and offensive speech, and the misalignment between academic target focus and real-world target prevalence. We finally argue that such low performance renders hate speech moderation with public detection models unfeasible, even in a human-in-the-loop setting which we find is prohibitively costly. Overall, we emphasize the need to evaluate future detection models from academia and platforms in real-world settings to address this global challenge.

本研究针对在线仇恨言论的检测性能在现实场景中存在的系统偏差问题，提出了HateDay，一个代表社交媒体设置的全球仇恨言论数据集，基于2022年9月21日发布的所有推文随机抽样。研究发现，现有学术数据集的评估方法高估了实际检测性能，尤其在非欧洲语言上表现较差，强调了未来检测模型在真实环境中评估的重要性。

HateDay：全球仇恨言论数据集的洞察