We present a human-and-model-in-the-loop process for dynamically generating
datasets and training better performing and more robust hate detection models.
We provide a new dataset of ~40,000 entries, generated and labelled by trained
annotators over four rounds of dynamic data creation. It includes ~15,000
challenging perturbations and each hateful entry has fine-grained labels for
the type and target of hate. Hateful entries make up 54% of the dataset, which
is substantially higher than comparable datasets. We show that model
performance is substantially improved using this approach. Models trained on
later rounds of data collection perform better on test sets and are harder for
annotators to trick. They also perform better on HateCheck, a suite of
functional tests for online hate detection. We provide the code, dataset and
annotation guidelines for other researchers to use. Accepted at ACL 2021.

本研究提出了一种利用人员与模型共同协作的动态数据集生成和训练模型的方法，生成了大约 40,000 项新数据并标注为 15,000 项令人挑战的扰动。通过这种方法训练出的模型性能大幅提升，具有更高的鲁棒性和更好的性能。