Model editing is a growing area focused on updating the knowledge embedded
within models. Among the various methodologies, ROME and MEMIT stand out as
leading "locate-and-edit" model editing techniques. While MEMIT enables batched
editing of memories, ROME is limited to changing one fact at a time. This paper
introduces a unifying framework that brings ROME and MEMIT under a single
conceptual umbrella, optimizing for the same goal, which we call the
"preservation-memorization" objective. This objective aims to preserve the
representations of certain selected vectors while memorizing the
representations of new factual information. Specifically, ROME optimizes this
objective using an equality constraint, whereas MEMIT employs a more flexible
least-square constraint. In addition to making batched edits, MEMIT also edits
the model at multiple layers. We disentangle the distribution of edits to
multiple layers from the optimization objective of MEMIT and show that these
edit-distribution algorithms should be considered separate entities worthy of
their own line of research.
Finally, we present EMMET - an Equality-constrained Mass Model Editing
algorithm for Transformers, a new batched memory-editing algorithm. With EMMET,
we present a closed form solution for the equality-constrained version of the
preservation-memorization objective. We show that EMMET is able to perform
batched-edits on par with MEMIT up to a batch-size of 256 and discuss the
challenges in stabilizing EMMET. By articulating the "locate-and-edit" model
editing algorithms under a simple conceptual framework of
"preservation-memorization", we aim to bridge the gap between intuition and
mathematics and hope to simplify the journey for future researchers in model
editing.

模型编辑是一个在更新模型中嵌入的知识方面不断发展的领域，本文针对 “定位和编辑” 模型编辑技术中的两种方法 ——ROME 和 MEMIT，提出了一个统一框架，将其优化为针对相同目标的 “保持记忆” 的目标，并提出了 EMMET 算法作为一种基于等式约束的批量记忆编辑算法。

模型编辑的统一框架

A Unified Framework for Model Editing

Recent work has shown exciting promise in updating large language models with
new memories, so as to replace obsolete information or add specialized
knowledge. However, this line of work is predominantly limited to updating
single associations. We develop MEMIT, a method for directly updating a
language model with many memories, demonstrating experimentally that it can
scale up to thousands of associations for GPT-J (6B) and GPT-NeoX (20B),
exceeding prior work by orders of magnitude. Our code and data are at
this https URL.

我们开发了 MEMIT 方法，直接更新带有许多记忆的语言模型，可以扩展到 GPT-J（6B）和 GPT-NeoX（20B）的成千上万的关联，超过以前的工作数个数量级。