Existing research on fairness evaluation of document classification models
mainly uses synthetic monolingual data without ground truth for author
demographic attributes. In this work, we assemble and publish a multilingual
Twitter corpus for the task of hate speech detection with inferred four author
demographic factors: age, country, gender and race/ethnicity. The corpus covers
five languages: English, Italian, Polish, Portuguese and Spanish. We evaluate
the inferred demographic labels with a crowdsourcing platform, Figure Eight. To
examine factors that can cause biases, we take an empirical analysis of
demographic predictability on the English corpus. We measure the performance of
four popular document classifiers and evaluate the fairness and bias of the
baseline classifiers on the author-level demographic attributes.

本研究到了关于文档分类模型公平性评估的现有研究，主要使用无作者人口属性基准事实的合成单语言数据。在该工作中，作者们汇编并发布了一个多语言 Twitter 语料库，用于推断作者的四个人口属性：年龄，国家，性别和种族 / 民族。该语料库涵盖了五种语言：英语、意大利语、波兰语、葡萄牙语和西班牙语。我们使用 Figure Eight 的众包平台评估了推断出的人口属性标签的准确性。为了检查可能引起偏见的因素，我们对英语语料库进行了人口预测的实证分析。我们评估了四种流行文档分类器的性能，并评估了基准分类器在作者级人口属性上的公平性和偏见。