Language models (LMs) derive their capabilities from extensive training on
diverse data, including potentially copyrighted material. These models can
memorize and generate content similar to their training data, posing potential
concerns. Therefore, model creators are motivated to develop mitigation methods
that prevent generating protected content. We term this procedure as copyright
takedowns for LMs, noting the conceptual similarity to (but legal distinction
from) the DMCA takedown This paper introduces the first evaluation of the
feasibility and side effects of copyright takedowns for LMs. We propose
CoTaEval, an evaluation framework to assess the effectiveness of copyright
takedown methods, the impact on the model's ability to retain uncopyrightable
factual knowledge from the training data whose recitation is embargoed, and how
well the model maintains its general utility and efficiency. We examine several
strategies, including adding system prompts, decoding-time filtering
interventions, and unlearning approaches. Our findings indicate that no tested
method excels across all metrics, showing significant room for research in this
unique problem setting and indicating potential unresolved challenges for live
policy proposals.

本文介绍了对语言模型版权撤回方法的可行性和副作用的首次评估，并提出了 CoTaEval 评估框架，以评估版权撤回方法的有效性、对模型保留非受版权限制的事实性知识的影响，以及模型在保持整体效用和效率方面的表现。通过添加系统提示、解码时的过滤干预和去学习方法等几种策略的研究，我们的发现表明，在所有指标上没有测试的方法表现出色，这表明在这个独特的问题设置中还有重要的研究空间，并暗示现行政策提议中可能存在未解决的挑战。