Large Transformer-based Pretrained Language Models (PLMs) dominate almost all
Natural Language Processing (NLP) tasks. Nevertheless, they still make mistakes
from time to time. For a model deployed in an industrial environment, fixing
these mistakes quickly and robustly is vital to improve user experiences.
Previous works formalize such problems as Model Editing (ME) and mostly focus
on fixing one mistake. However, the one-mistake-fixing scenario is not an
accurate abstraction of the real-world challenge. In the deployment of AI
services, there are ever-emerging mistakes, and the same mistake may recur if
not corrected in time. Thus a preferable solution is to rectify the mistakes as
soon as they appear nonstop. Therefore, we extend the existing ME into
Sequential Model Editing (SME) to help develop more practical editing methods.
Our study shows that most current ME methods could yield unsatisfying results
in this scenario. We then introduce Transformer-Patcher, a novel model editor
that can shift the behavior of transformer-based models by simply adding and
training a few neurons in the last Feed-Forward Network layer. Experimental
results on both classification and generation tasks show that
Transformer-Patcher can successively correct up to thousands of errors
(Reliability) and generalize to their equivalent inputs (Generality) while
retaining the model's accuracy on irrelevant inputs (Locality). Our method
outperforms previous fine-tuning and HyperNetwork-based methods and achieves
state-of-the-art performance for Sequential Model Editing (SME). The code is
available at this https URL.

本研究提出一种被称为 Transformer-Patcher 的神经网络模型，能够通过简单地添加和训练最后一层前馈网络中的少量神经元，连续纠正长序列中的错误，达到了顺序模型编辑（SME）的最优表现，解决了工业环境中部署的模型如何快速准确地修正错误问题。

Transformer-Patcher: 一错必补的神经元

Transformer-Patcher: One Mistake worth One Neuron

Pre-trained language models learn large amounts of knowledge from their
training corpus, while the memorized facts could become outdated over a few
years. Model editing aims to make post-hoc updates on specific facts in a model
while leaving irrelevant knowledge unchanged. However, existing work studies
only the monolingual scenario. In this paper, we focus on cross-lingual model
editing. Firstly, we propose the definition and metrics of the cross-lingual
model editing, where updates in a single language should take effect in the
others as well. Next, we propose a simple framework to convert a monolingual
model editing approach to its cross-lingual variant using the parallel corpus.
Experiments show that such an approach outperforms monolingual baselines by a
large margin. Furthermore, we propose language anisotropic editing to improve
cross-lingual editing by estimating parameter importance for each language.
Experiments reveal that language anisotropic editing decreases the editing
failing rate by another $26\%$ relatively.

本文提出了适用于跨语言模型的模型编辑方法，使用平行语料库实现了单语言模型编辑方法的跨语言变体，并采用了语言各向异性编辑方法，实现了显着的编辑率下降。

语言向异性跨语言模型编辑

Language Anisotropic Cross-Lingual Model Editing

We analyze the storage and recall of factual associations in autoregressive
transformer language models, finding evidence that these associations
correspond to localized, directly-editable computations. We first develop a
causal intervention for identifying neuron activations that are decisive in a
model's factual predictions. This reveals a distinct set of steps in
middle-layer feed-forward modules that mediate factual predictions while
processing subject tokens. To test our hypothesis that these computations
correspond to factual association recall, we modify feed-forward weights to
update specific factual associations using Rank-One Model Editing (ROME). We
find that ROME is effective on a standard zero-shot relation extraction (zsRE)
model-editing task, comparable to existing methods. To perform a more sensitive
evaluation, we also evaluate ROME on a new dataset of counterfactual
assertions, on which it simultaneously maintains both specificity and
generalization, whereas other methods sacrifice one or another. Our results
confirm an important role for mid-layer feed-forward modules in storing factual
associations and suggest that direct manipulation of computational mechanisms
may be a feasible approach for model editing. The code, dataset,
visualizations, and an interactive demo notebook are available at
this https URL

本文使用因果干预技术研究了自回归转换语言模型中实际关联的存储和检索，并发现这些关联对应于本地化的可直接编辑的计算。研究表明中间层前馈模块在存储实际关联方面具有重要作用，并且为模型编辑提供了直接操作计算机制的方法。