Large Language Models (LLMs) often memorize sensitive, private, or
copyrighted data during pre-training. LLM unlearning aims to eliminate the
influence of undesirable data from the pre-trained model while preserving the
model's utilities on other tasks. Several practical methods have recently been
proposed for LLM unlearning, mostly based on gradient ascent (GA) on the loss
of undesirable data. However, on certain unlearning tasks, these methods either
fail to effectively unlearn the target data or suffer from catastrophic
collapse -- a drastic degradation of the model's utilities.
In this paper, we propose Negative Preference Optimization (NPO), a simple
alignment-inspired method that could efficiently and effectively unlearn a
target dataset. We theoretically show that the progression toward catastrophic
collapse by minimizing the NPO loss is exponentially slower than GA. Through
experiments on synthetic data and the benchmark TOFU dataset, we demonstrate
that NPO-based methods achieve a better balance between unlearning the
undesirable data and maintaining the model's utilities. We also observe that
NPO-based methods generate more sensible outputs than GA-based methods, whose
outputs are often gibberish. Remarkably, on TOFU, NPO-based methods are the
first to achieve reasonable unlearning results in forgetting 50% (or more) of
the training data, whereas existing methods already struggle with forgetting
10% of training data.

大型语言模型（LLMs）在预训练过程中经常记忆敏感、私人或受版权保护的数据。LLM unlearning 旨在消除预训练模型中不需要的数据的影响，同时保留模型在其他任务上的效用。本文提出了一种简单的基于对齐的方法 ——Negative Preference Optimization（NPO），可以高效有效地 unlearn 目标数据集。通过在合成数据和基准 TOFU 数据集上的实验证明，基于 NPO 的方法在 unlearn 不需要的数据和保持模型效用之间取得了更好的平衡。我们还观察到，与输出通常为无意义的 GA-based 方法相比，基于 NPO 的方法生成更有意义的输出。值得注意的是，在 TOFU 上，基于 NPO 的方法在忘记 50%（或更多）训练数据方面取得了合理的 unlearning 结果，而现有的方法已经在忘记 10% 的训练数据方面存在困难。