Large language models (LLMs) inevitably memorize sensitive, copyrighted, and
harmful knowledge from the training corpus; therefore, it is crucial to erase
this knowledge from the models. Machine unlearning is a promising solution for
efficiently removing specific knowledge by post hoc modifying models. In this
paper, we propose a Real-World Knowledge Unlearning benchmark (RWKU) for LLM
unlearning. RWKU is designed based on the following three key factors: (1) For
the task setting, we consider a more practical and challenging unlearning
setting, where neither the forget corpus nor the retain corpus is accessible.
(2) For the knowledge source, we choose 200 real-world famous people as the
unlearning targets and show that such popular knowledge is widely present in
various LLMs. (3) For the evaluation framework, we design the forget set and
the retain set to evaluate the model's capabilities across various real-world
applications. Regarding the forget set, we provide four four membership
inference attack (MIA) methods and nine kinds of adversarial attack probes to
rigorously test unlearning efficacy. Regarding the retain set, we assess
locality and utility in terms of neighbor perturbation, general ability,
reasoning ability, truthfulness, factuality, and fluency. We conduct extensive
experiments across two unlearning scenarios, two models and six baseline
methods and obtain some meaningful findings. We release our benchmark and code
publicly at this http URL for future work.

机器遗忘是一种有效地通过事后修改模型来消除特定知识的方法，在本文中，我们提出了一个用于大型语言模型（LLMs）遗忘的真实世界知识遗忘基准（RWKU），该基准考虑了任务设置、知识来源和评估框架等关键因素。