How can we detect if copyrighted content was used in the training process of
a language model, considering that the training data is typically undisclosed?
We are motivated by the premise that a language model is likely to identify
verbatim excerpts from its training text. We propose DE-COP, a method to
determine whether a piece of copyrighted content was included in training.
DE-COP's core approach is to probe an LLM with multiple-choice questions, whose
options include both verbatim text and their paraphrases. We construct
BookTection, a benchmark with excerpts from 165 books published prior and
subsequent to a model's training cutoff, along with their paraphrases. Our
experiments show that DE-COP surpasses the prior best method by 9.6% in
detection performance (AUC) on models with logits available. Moreover, DE-COP
also achieves an average accuracy of 72% for detecting suspect books on fully
black-box models where prior methods give $\approx$ 4% accuracy. Our code and
datasets are available at this https URL

我们提出了 DE-COP 方法，通过向语言模型提出包含直接引文和释义的多项选择题，来确定受版权保护的内容是否包含在训练中。我们构建了 BookTection 基准测试集，其中包含了 165 本在模型训练截止日期之前和之后出版的书籍摘录及其释义。我们的实验结果显示，DE-COP 方法在具有 logits 可用的模型上比之前的最佳方法的检测性能（AUC）提高了 9.6%。此外，DE-COP 方法在完全黑盒模型上的检测可疑书籍的平均准确率为 72%，而之前的方法只能达到约 4% 的准确率。