Since state-of-the-art approaches to offensive language detection rely on
supervised learning, it is crucial to quickly adapt them to the continuously
evolving scenario of social media. While several approaches have been proposed
to tackle the problem from an algorithmic perspective, so to reduce the need
for annotated data, less attention has been paid to the quality of these data.
Following a trend that has emerged recently, we focus on the level of agreement
among annotators while selecting data to create offensive language datasets, a
task involving a high level of subjectivity. Our study comprises the creation
of three novel datasets of English tweets covering different topics and having
five crowd-sourced judgments each. We also present an extensive set of
experiments showing that selecting training and test data according to
different levels of annotators' agreement has a strong effect on classifiers
performance and robustness. Our findings are further validated in cross-domain
experiments and studied using a popular benchmark dataset. We show that such
hard cases, where low agreement is present, are not necessarily due to
poor-quality annotation and we advocate for a higher presence of ambiguous
cases in future datasets, particularly in test sets, to better account for the
different points of view expressed online.

本文提出了一种用于创建自然语言数据集的方法，通过将标注者的观点选为不同的训练集和测试集，可以提高分类器的性能和鲁棒性，为社交媒体中的恶意语言检测提供更好的数据准备。