In this paper, we explore the following question: how far are we from real
synonym substitution attacks (SSAs). We approach this question by examining how
SSAs replace words in the original sentence and show that there are still
unresolved obstacles that make current SSAs generate invalid adversarial
samples. We reveal that four widely used word substitution methods generate a
large fraction of invalid substitution words that are ungrammatical or do not
preserve the original sentence's semantics. Next, we show that the semantic and
grammatical constraints used in SSAs for detecting invalid word replacements
are highly insufficient in detecting invalid adversarial samples. Our work is
an important stepping stone to constructing better SSAs in the future.

本研究探讨了同义词替换攻击（SSAs）的实际可行性，并揭示当前 SSA 中存在的未解决问题，包括四种常用的替换方法产生大量无效替换单词，语法及语义约束不足以检测到无效的样本，该研究为今后构建更好的 SSAs 提供了重要的契机。

真正的同义词替换攻击有多远？

How Far Are We from Real Synonym Substitution Attacks?

State-of-the-art attacks on NLP models lack a shared definition of a what
constitutes a successful attack. We distill ideas from past work into a unified
framework: a successful natural language adversarial example is a perturbation
that fools the model and follows some linguistic constraints. We then analyze
the outputs of two state-of-the-art synonym substitution attacks. We find that
their perturbations often do not preserve semantics, and 38% introduce
grammatical errors. Human surveys reveal that to successfully preserve
semantics, we need to significantly increase the minimum cosine similarities
between the embeddings of swapped words and between the sentence encodings of
original and perturbed sentences.With constraints adjusted to better preserve
semantics and grammaticality, the attack success rate drops by over 70
percentage points.

通过分析最先进程序的同义词替换攻击，本文提出了成功自然语言攻击的定义 —— 伪造的文本必须欺骗模型、遵守一些语言约束，并发现其扰动往往不能保持语义，而人类调查表明交换词汇的嵌入和原始 / 扰动句子的句子编码之间的最小余弦相似性应显著提高以保持有效的语义性和语法性，一旦应用了相应的限制成功率会下降 70%。