Recently, machine unlearning, which seeks to erase specific data stored in
the pre-trained or fine-tuned models, has emerged as a crucial protective
measure for LLMs. However, unlearning approaches for LLMs that have been
considered thus far have focused on the removal of independent data points and
have not taken into account that the stored facts are logically connected to
one another and form an implicit knowledge graph. To facilitate the development
of structural unlearning methods, which are essential for the practical
application of unlearning, we propose PISTOL, a pipeline for compiling
multi-scenario datasets for benchmarking structural LLM unlearning.
Additionally, leveraging sample datasets synthesized using PISTOL, we conducted
benchmarks with four distinct unlearning methods on both Llama2-7B and
Mistral-7B models. This analysis helps to illustrate the prevailing challenges
in effectively and robustly removing highly inter-connected data, batched data,
or data skewed towards a specific domain. It also highlights the choice of
pre-trained model can impact unlearning performance. This work not only
advances our understandings on the limitation of current LLMs unlearning
methods and proposes future research directions, but also provides a replicable
framework for ongoing exploration and validation in the field.

为了推动现有 LLMs 遗忘方法的发展并解决其局限性，本文提出了 PISTOL 管道来编制多场景数据集用于基准测试结构化 LLMs 遗忘，并使用 PISTOL 生成的样本数据集进行了四种不同遗忘方法在 Llama2-7B 和 Mistral-7B 模型上的基准测试，以揭示在有效和鲁棒地移除高度相互连接的数据、批处理数据或偏向特定领域的数据时所面临的主要挑战，并强调预训练模型的选择如何影响遗忘效果。该工作不仅推动了我们对当前 LLMs 遗忘方法的局限性的理解，并提出了未来的研究方向，还为领域内的持续探索和验证提供了可复制的框架。