Recent literature has highlighted potential risks to academic integrity
associated with large language models (LLMs), as they can memorize parts of
training instances and reproduce them in the generated texts without proper
attribution. In addition, given their capabilities in generating high-quality
texts, plagiarists can exploit LLMs to generate realistic paraphrases or
summaries indistinguishable from original work. In response to possible
malicious use of LLMs in plagiarism, we introduce PlagBench, a comprehensive
dataset consisting of 46.5K synthetic plagiarism cases generated using three
instruction-tuned LLMs across three writing domains. The quality of PlagBench
is ensured through fine-grained automatic evaluation for each type of
plagiarism, complemented by human annotation. We then leverage our proposed
dataset to evaluate the plagiarism detection performance of five modern LLMs
and three specialized plagiarism checkers. Our findings reveal that GPT-3.5
tends to generates paraphrases and summaries of higher quality compared to
Llama2 and GPT-4. Despite LLMs' weak performance in summary plagiarism
identification, they can surpass current commercial plagiarism detectors.
Overall, our results highlight the potential of LLMs to serve as robust
plagiarism detection tools.

最近的文献强调了大型语言模型（LLMs）与学术诚信相关的潜在风险，它们可以记忆部分训练实例并在生成的文本中无妥善归属地复制。此外，鉴于它们在生成高质量文本方面的能力，剽窃者可以利用 LLMs 生成与原作无法区分的逼真释义或摘要。为了应对 LLMs 可能在剽窃行为中的恶意使用，我们介绍了 PlagBench，这是一个综合的数据集，由三个针对不同写作领域的三个指导调整的 LLMs 生成的 46.5K 个合成剽窃案例组成。通过对每种类型的剽窃进行细粒度的自动评估和人工注释来确保 PlagBench 的质量。然后，我们利用我们提出的数据集来评估五个现代 LLMs 和三个专门的剽窃检测器的剽窃检测性能。我们的研究结果表明，与 Llama2 和 GPT-4 相比，GPT-3.5 倾向于生成更高质量的释义和摘要。尽管 LLMs 在摘要剽窃识别方面表现较差，但它们可以超过当前的商业剽窃检测器。总体而言，我们的结果突显了 LLMs 作为强大剽窃检测工具的潜力。