To detect the deployment of large language models for malicious use cases
(e.g., fake content creation or academic plagiarism), several approaches have
recently been proposed for identifying AI-generated text via watermarks or
statistical irregularities. How robust are these detection algorithms to
paraphrases of AI-generated text? To stress test these detectors, we first
train an 11B parameter paraphrase generation model (DIPPER) that can paraphrase
paragraphs, optionally leveraging surrounding text (e.g., user-written prompts)
as context. DIPPER also uses scalar knobs to control the amount of lexical
diversity and reordering in the paraphrases. Paraphrasing text generated by
three large language models (including GPT3.5-davinci-003) with DIPPER
successfully evades several detectors, including watermarking, GPTZero,
DetectGPT, and OpenAI's text classifier. For example, DIPPER drops the
detection accuracy of DetectGPT from 70.3% to 4.6% (at a constant false
positive rate of 1%), without appreciably modifying the input semantics. To
increase the robustness of AI-generated text detection to paraphrase attacks,
we introduce a simple defense that relies on retrieving semantically-similar
generations and must be maintained by a language model API provider. Given a
candidate text, our algorithm searches a database of sequences previously
generated by the API, looking for sequences that match the candidate text
within a certain threshold. We empirically verify our defense using a database
of 15M generations from a fine-tuned T5-XXL model and find that it can detect
80% to 97% of paraphrased generations across different settings, while only
classifying 1% of human-written sequences as AI-generated. We will open source
our code, model and data for future research.

通过语义相似的生成物和检测算法，该研究提出了一种简单的防御，以增加对 AI 生成文本的检测器对抗改写攻击的鲁棒性。