Pre-trained Large Language Models (LLMs) have demonstrated remarkable
capabilities but also pose risks by learning and generating copyrighted
material, leading to significant legal and ethical concerns. To address these
issues, it is critical for model owners to be able to unlearn copyrighted
content at various time steps. We explore the setting of sequential unlearning,
where copyrighted content is removed over multiple time steps - a scenario that
has not been rigorously addressed. To tackle this challenge, we propose Stable
Sequential Unlearning (SSU), a novel unlearning framework for LLMs, designed to
have a more stable process to remove copyrighted content from LLMs throughout
different time steps using task vectors, by incorporating additional random
labeling loss and applying gradient-based weight saliency mapping. Experiments
demonstrate that SSU finds a good balance between unlearning efficacy and
maintaining the model's general knowledge compared to existing baselines.

在这篇论文中，我们提出了一种稳定的序列遗忘（SSU）框架，该框架使用任务向量，通过引入额外的随机标签损失和应用基于梯度的权重显著性映射，以更稳定的方式在不同的时间步骤中从 LLMs 中删除受版权保护的内容，实验证明 SSU 在遗忘效果和保持模型的一般知识之间取得了良好的平衡。

通过机器遗忘来避免版权侵权

Avoiding Copyright Infringement via Machine Unlearning

Pretrained Language Models (LMs) memorize a vast amount of knowledge during
initial pretraining, including information that may violate the privacy of
personal lives and identities. Previous work addressing privacy issues for
language models has mostly focused on data preprocessing and differential
privacy methods, both requiring re-training the underlying LM. We propose
knowledge unlearning as an alternative method to reduce privacy risks for LMs
post hoc. We show that simply performing gradient ascent on target token
sequences is effective at forgetting them with little to no degradation of
general language modeling performances for larger LMs; it sometimes even
substantially improves the underlying LM with just a few iterations. We also
find that sequential unlearning is better than trying to unlearn all the data
at once and that unlearning is highly dependent on which kind of data (domain)
is forgotten. By showing comparisons with a previous data preprocessing method
and a decoding method known to mitigate privacy risks for LMs, we show that
unlearning can give a stronger empirical privacy guarantee in scenarios where
the data vulnerable to extraction attacks are known a priori while being much
more efficient and robust. We release the code and dataset needed to replicate
our results at this https URL.

本研究提出知识遗忘（knowledge unlearning）作为一种减少预训练语言模型（pretrained language models）隐私风险的方法，通过对目标标记序列进行梯度上升来遗忘它们，并发现顺序遗忘优于尝试一次性遗忘所有数据，对于特定领域的数据，知识遗忘具有更强的实证隐私保证，同时更加高效和鲁棒。