Large language models (LLMs) require continual knowledge updates to stay
abreast of the ever-changing world facts, prompting the formulation of lifelong
model editing task. While recent years have witnessed the development of
various techniques for single and batch editing, these methods either fail to
apply or perform sub-optimally when faced with lifelong editing. In this paper,
we introduce LEMoE, an advanced Mixture of Experts (MoE) adaptor for lifelong
model editing. We first analyze the factors influencing the effectiveness of
conventional MoE adaptor in lifelong editing, including catastrophic
forgetting, inconsistent routing and order sensitivity. Based on these
insights, we propose a tailored module insertion method to achieve lifelong
editing, incorporating a novel KV anchor routing to enhance routing consistency
between training and inference stage, along with a concise yet effective
clustering-based editing order planning. Experimental results demonstrate the
effectiveness of our method in lifelong editing, surpassing previous model
editing techniques while maintaining outstanding performance in batch editing
task. Our code will be available.

LLMs require continual knowledge updates, and this paper introduces LEMoE, an advanced MoE adaptor for lifelong model editing, addressing issues like catastrophic forgetting and inconsistent routing, with experimental results demonstrating its effectiveness.