The existing fine-tuning paradigm for language models is brittle in knowledge
editing scenarios, where the model must incorporate new information without
extensive retraining. This brittleness often results in overfitting, reduced
performance, and unnatural language generation. To address this, we propose
Consistent In-Context Editing (ICE), a novel approach that leverages the
model's in-context learning capability to tune toward a contextual distribution
rather than a one-hot target. ICE introduces a straightforward optimization
framework that includes both a target and a procedure, enhancing the robustness
and effectiveness of gradient-based tuning methods. We provide analytical
insights into ICE across four critical aspects of knowledge editing: accuracy,
locality, generalization, and linguistic quality, showing its advantages.
Experimental results across four datasets confirm the effectiveness of ICE and
demonstrate its potential for continual editing, ensuring that updated
information is incorporated while preserving the integrity of the model.

已有的语言模型微调范式在知识编辑场景下容易出现脆弱性，本文提出了一种名为一致上下文编辑（ICE）的新方法，利用模型的上下文学习能力来调整模型以适应上下文分布而非单一目标，通过直观的优化框架增强了梯度调整方法的鲁棒性和效果，实验证明了 ICE 的优势及其在不断编辑中的潜力，确保了更新的信息被纳入同时保持模型的完整性。