A new approach to linguistic watermarking of language models is presented in
which information is imperceptibly inserted into the output text while
preserving its readability and original meaning. A cross-attention mechanism is
used to embed watermarks in the text during inference. Two methods using
cross-attention are presented that minimize the effect of watermarking on the
performance of a pretrained model. Exploration of different training strategies
for optimizing the watermarking and of the challenges and implications of
applying this approach in real-world scenarios clarified the tradeoff between
watermark robustness and text quality. Watermark selection substantially
affects the generated output for high entropy sentences. This proactive
watermarking approach has potential application in future model development.

语言模型的语言水印技术通过交叉注意力机制在输出文本中嵌入信息，保持其可读性和原始含义，本文提出了两种使用交叉注意力的方法，最小化对预训练模型性能的影响。研究了优化水印技术的不同训练策略以及在实际应用中的挑战和影响，阐明了水印鲁棒性和文本质量之间的权衡，高熵句子的水印选择显著影响生成的输出，这种主动水印技术在未来模型开发中具有潜在的应用。