Detecting whether copyright holders' works were used in LLM pretraining is
poised to be an important problem. This work proposes using data watermarks to
enable principled detection with only black-box model access, provided that the
rightholder contributed multiple training documents and watermarked them before
public release. By applying a randomly sampled data watermark, detection can be
framed as hypothesis testing, which provides guarantees on the false detection
rate. We study two watermarks: one that inserts random sequences, and another
that randomly substitutes characters with Unicode lookalikes. We first show how
three aspects of watermark design -- watermark length, number of duplications,
and interference -- affect the power of the hypothesis test. Next, we study how
a watermark's detection strength changes under model and dataset scaling: while
increasing the dataset size decreases the strength of the watermark, watermarks
remain strong if the model size also increases. Finally, we view SHA hashes as
natural watermarks and show that we can robustly detect hashes from
BLOOM-176B's training data, as long as they occurred at least 90 times.
Together, our results point towards a promising future for data watermarks in
real world use.

利用数据水印将版权持有者的作品用于 LLM 预训练进行检测，可作为一个重要的问题。我们研究了使用数据水印进行便于黑箱模型访问的有原则的检测的方法。通过应用随机抽样的数据水印，我们将检测问题构建为假设检验，从而提供了关于虚警率的保证。我们分析了水印的三个方面 —— 水印长度、复制次数和干扰，对假设检验的能力产生了影响。同时研究了水印在模型和数据集缩放下的检测力变化：虽然增加数据集规模会降低水印的检测力，但如果模型大小也增加，水印仍然具有一定的强度。最后，我们将 SHA 哈希视为自然水印，并展示了我们能够鲁棒地检测来自 BLOOM-176B 的训练数据中的哈希，只要它们出现了至少 90 次。综上，我们的结果指向了数据水印在实际应用中具有广阔前景。