This study investigates the concept of the `right to be forgotten' within the
context of large language models (LLMs). We explore machine unlearning as a
pivotal solution, with a focus on pre-trained models--a notably
under-researched area. Our research delineates a comprehensive framework for
machine unlearning in pre-trained LLMs, encompassing a critical analysis of
seven diverse unlearning methods. Through rigorous evaluation using curated
datasets from arXiv, books, and GitHub, we establish a robust benchmark for
unlearning performance, demonstrating that these methods are over $10^5$ times
more computationally efficient than retraining. Our results show that
integrating gradient ascent with gradient descent on in-distribution data
improves hyperparameter robustness. We also provide detailed guidelines for
efficient hyperparameter tuning in the unlearning process. Our findings advance
the discourse on ethical AI practices, offering substantive insights into the
mechanics of machine unlearning for pre-trained LLMs and underscoring the
potential for responsible AI development.

通过对预训练大语言模型的机器遗忘进行综合研究，我们探讨了‘被遗忘权’的概念，重点关注了尚未充分研究的预训练模型领域。通过对来自 arXiv、书籍和 GitHub 的精选数据集进行严格评估，我们建立了机器遗忘性能的鲁棒基准，并证明这些方法的计算效率比重新训练高出 10^5 倍。我们的研究结果表明，在分布数据上将梯度上升与梯度下降相结合可以提高超参数的稳健性。此外，我们还提供了高效超参数调整的详细指南。我们的发现推动了关于道德 AI 实践的讨论，为预训练大语言模型的机器遗忘机制提供了实质性的见解，并强调了负责任的 AI 发展的潜力。

预训练大型语言模型的机器消除学习

Machine Unlearning of Pre-trained Large Language Models

The emergence and growth of research on issues of ethics in AI, and in
particular algorithmic fairness, has roots in an essential observation that
structural inequalities in society are reflected in the data used to train
predictive models and in the design of objective functions. While research
aiming to mitigate these issues is inherently interdisciplinary, the design of
unbiased algorithms and fair socio-technical systems are key desired outcomes
which depend on practitioners from the fields of data science and computing.
However, these computing fields broadly also suffer from the same
under-representation issues that are found in the datasets we analyze. This
disconnect affects the design of both the desired outcomes and metrics by which
we measure success. If the ethical AI research community accepts this, we
tacitly endorse the status quo and contradict the goals of non-discrimination
and equity which work on algorithmic fairness, accountability, and transparency
seeks to address. Therefore, we advocate in this work for diversifying
computing as a core priority of the field and our efforts to achieve ethical AI
practices. We draw connections between the lack of diversity within academic
and professional computing fields and the type and breadth of the biases
encountered in datasets, machine learning models, problem formulations, and
interpretation of results. Examining the current fairness/ethics in AI
literature, we highlight cases where this lack of diverse perspectives has been
foundational to the inequity in treatment of underrepresented and protected
group data. We also look to other professional communities, such as in law and
health, where disparities have been reduced both in the educational diversity
of trainees and among professional practices. We use these lessons to develop
recommendations that provide concrete steps for the computing community to
increase diversity.

本文提倡将多样性作为核心优先事项，以实现算法公平所追求的不歧视和公正目标，并为计算机科学的从业人员提供具体的多样性措施建议，以增加多样性，并提高算法公平的实践水平。