Model editing aims to efficiently alter the behavior of Large Language Models
(LLMs) within a desired scope, while ensuring no adverse impact on other
inputs. Recent years have witnessed various model editing methods been
proposed. However, these methods either exhibit poor overall performance or
struggle to strike a balance between generalization and locality. We propose
MOMoE, a model editing adapter utilizing a Mixture of Experts (MoE)
architecture with a knowledge anchor routing strategy. MOMoE updates knowledge
using a bypass MoE structure, keeping the original parameters unchanged to
preserve the general ability of LLMs. And, the knowledge anchor routing ensures
that inputs requiring similar knowledge are routed to the same expert, thereby
enhancing the generalization of the updated knowledge. Experimental results
show the superiority of our approach over both batch editing and sequential
batch editing tasks, exhibiting exceptional overall performance alongside
outstanding balance between generalization and locality. Our code will be
available.

MOMoE 是一种模型编辑适配器，采用专家混合（MoE）架构和知识锚定路由策略，以高效地改变大型语言模型（LLMs）的行为，并确保不对其他输入产生负面影响。实验结果表明，MOMoE 方法在整体性能和泛化能力之间展现出卓越的平衡，优于批量编辑和连续批量编辑任务。