Toxic content detection aims to identify content that can offend or harm its
recipients. Automated classifiers of toxic content need to be robust against
adversaries who deliberately try to bypass filters. We propose a method of
generating realistic model-agnostic attacks using a lexicon of toxic tokens,
which attempts to mislead toxicity classifiers by diluting the toxicity signal
either by obfuscating toxic tokens through character-level perturbations, or by
injecting non-toxic distractor tokens. We show that these realistic attacks
reduce the detection recall of state-of-the-art neural toxicity detectors,
including those using ELMo and BERT, by more than 50% in some cases. We explore
two approaches for defending against such attacks. First, we examine the effect
of training on synthetically noised data. Second, we propose the Contextual
Denoising Autoencoder (CDAE): a method for learning robust representations that
uses character-level and contextual information to denoise perturbed tokens. We
show that the two approaches are complementary, improving robustness to both
character-level perturbations and distractors, recovering a considerable
portion of the lost accuracy. Finally, we analyze the robustness
characteristics of the most competitive methods and outline practical
considerations for improving toxicity detectors.

本文提出了一种通过生成包含毒性单词的对抗样本，从而检验模型对毒性内容的识别性能，并探讨了使用 Contextual Denoising Autoencoder 提高模型鲁棒性的方法。实验表明，该方法在某些情况下可以将现有模型的检测准确率降低 50％以上。