Adversarial examples in NLP are receiving increasing research attention. One
line of investigation is the generation of word-level adversarial examples
against fine-tuned Transformer models that preserve naturalness and
grammaticality. Previous work found that human- and machine-generated
adversarial examples are comparable in their naturalness and grammatical
correctness. Most notably, humans were able to generate adversarial examples
much more effortlessly than automated attacks. In this paper, we provide a
detailed analysis of exactly how humans create these adversarial examples. By
exploring the behavioural patterns of human workers during the generation
process, we identify statistically significant tendencies based on which words
humans prefer to select for adversarial replacement (e.g., word frequencies,
word saliencies, sentiment) as well as where and when words are replaced in an
input sequence. With our findings, we seek to inspire efforts that harness
human strategies for more robust NLP models.

本研究分析了人类如何生成针对细调 Transformer 模型的自然性和语法正确性保持的词级对抗性示例，通过探索人类工作者在生成过程中的行为模式，我们识别出人类更喜欢选择哪些单词进行对抗替换以及何时何地进行替换。研究结果可用于启发利用人类策略制定更强大的自然语言处理模型。