To mitigate the potential misuse of large language models (LLMs), recent
research has developed watermarking algorithms, which restrict the generation
process to leave an invisible trace for watermark detection. Due to the
two-stage nature of the task, most studies evaluate the generation and
detection separately, thereby presenting a challenge in unbiased, thorough, and
applicable evaluations. In this paper, we introduce WaterBench, the first
comprehensive benchmark for LLM watermarks, in which we design three crucial
factors: (1) For \textbf{benchmarking procedure}, to ensure an apples-to-apples
comparison, we first adjust each watermarking method's hyper-parameter to reach
the same watermarking strength, then jointly evaluate their generation and
detection performance. (2) For \textbf{task selection}, we diversify the input
and output length to form a five-category taxonomy, covering $9$ tasks. (3) For
\textbf{evaluation metric}, we adopt the GPT4-Judge for automatically
evaluating the decline of instruction-following abilities after watermarking.
We evaluate $4$ open-source watermarks on $2$ LLMs under $2$ watermarking
strengths and observe the common struggles for current methods on maintaining
the generation quality. The code and data are available at
https://github.com/THU-KEG/WaterBench.

为了解决大型语言模型的滥用问题，该研究开发了水印算法，通过限制生成过程以留下不可见痕迹进行水印检测。本文引入了 WaterBench，第一个针对大型语言模型水印的全面基准测试，通过设计三个关键要素进行了综合评估，包括基准测试过程、任务选择和评估指标，观察到当前方法在保持生成质量方面存在共同困难。