Watermarking generative-AI systems, such as LLMs, has gained considerable
interest, driven by their enhanced capabilities across a wide range of tasks.
Although current approaches have demonstrated that small, context-dependent
shifts in the word distributions can be used to apply and detect watermarks,
there has been little work in analyzing the impact that these perturbations
have on the quality of generated texts. Balancing high detectability with
minimal performance degradation is crucial in terms of selecting the
appropriate watermarking setting; therefore this paper proposes a simple
analysis framework where comparative assessment, a flexible NLG evaluation
framework, is used to assess the quality degradation caused by a particular
watermark setting. We demonstrate that our framework provides easy
visualization of the quality-detection trade-off of watermark settings,
enabling a simple solution to find an LLM watermark operating point that
provides a well-balanced performance. This approach is applied to two different
summarization systems and a translation system, enabling cross-model analysis
for a task, and cross-task analysis.

通过比较评估的灵活自然语言生成评估框架，本研究提出了一个简单的分析框架，用于评估特定水印设置引起的生成文本质量降低，以找到平衡性能和易检测性的最佳水印操作点。该方法应用于两个不同的摘要系统和一个翻译系统，实现了任务内和跨任务的模型分析。