Recently, while large language models (LLMs) have demonstrated impressive
results, they still suffer from hallucination, i.e., the generation of false
information. Model editing is the task of fixing factual mistakes in LLMs; yet,
most previous works treat it as a one-time task, paying little attention to
ever-emerging mistakes generated by LLMs. We address the task of sequential
model editing (SME) that aims to rectify mistakes continuously. A Dynamic
Auxiliary Fusion Network (DAFNet) is designed to enhance the semantic
interaction among the factual knowledge within the entire sequence, preventing
catastrophic forgetting during the editing process of multiple knowledge
triples. Specifically, (1) for semantic fusion within a relation triple, we
aggregate the intra-editing attention flow into auto-regressive self-attention
with token-level granularity in LLMs. We further leverage multi-layer diagonal
inter-editing attention flow to update the weighted representations of the
entire sequence-level granularity. (2) Considering that auxiliary parameters
are required to store the knowledge for sequential editing, we construct a new
dataset named \textbf{DAFSet}, fulfilling recent, popular, long-tail and robust
properties to enhance the generality of sequential editing. Experiments show
DAFNet significantly outperforms strong baselines in single-turn and sequential
editing. The usage of DAFSet also consistently improves the performance of
other auxiliary network-based methods in various scenarios

大型语言模型中的模型编辑任务，设计了动态辅助融合网络（DAFNet）以提升语义交互，并使用新构建的数据集 DAFSet 进行连续编辑，实验证明 DAFNet 在单轮和连续编辑任务中显著优于其他方法。

DAFNet：大语言模型中序贴模型编辑的动态辅助融合

DAFNet: Dynamic Auxiliary Fusion for Sequential Model Editing in Large  Language Models

This study presents a targeted model editing analysis focused on the latest
large language model, Llama-3. We explore the efficacy of popular model editing
techniques - ROME, MEMIT, and EMMET, which are designed for precise layer
interventions. We identify the most effective layers for targeted edits through
an evaluation that encompasses up to 4096 edits across three distinct
strategies: sequential editing, batch editing, and a hybrid approach we call as
sequential-batch editing. Our findings indicate that increasing edit
batch-sizes may degrade model performance more significantly than using smaller
edit batches sequentially for equal number of edits. With this, we argue that
sequential model editing is an important component for scaling model editing
methods and future research should focus on methods that combine both batched
and sequential editing. This observation suggests a potential limitation in
current model editing methods which push towards bigger edit batch sizes, and
we hope it paves way for future investigations into optimizing batch sizes and
model editing performance.

本研究重点针对最新的大型语言模型 Llama-3 进行了定向模型编辑分析。我们探讨了针对精确层次干预设计的流行模型编辑技术 ——ROME、MEMIT 和 EMMET 的有效性。通过对三种不同策略（顺序编辑、批次编辑和顺序 - 批次编辑）的 4096 次编辑进行评估，我们确定了最有效的编辑层次。我们的发现表明，增加编辑批次大小可能会比使用较小的编辑批次按顺序执行相同数量的编辑更明显地降低模型性能。基于此，我们认为顺序模型编辑在扩展模型编辑方法方面是一个重要组成部分，并且未来的研究应该专注于将批量编辑和顺序编辑相结合的方法。这一观察表明了当前模型编辑方法在推动更大的编辑批次大小方面存在潜在限制，我们希望这为优化批次大小和模型编辑性能的未来研究铺平道路。