The factual knowledge acquired during pre-training and stored in the
parameters of Language Models (LMs) can be useful in downstream tasks (e.g.,
question answering or textual inference). However, some facts can be
incorrectly induced or become obsolete over time. We present KnowledgeEditor, a
method which can be used to edit this knowledge and, thus, fix 'bugs' or
unexpected predictions without the need for expensive re-training or
fine-tuning. Besides being computationally efficient, KnowledgeEditordoes not
require any modifications in LM pre-training (e.g., the use of meta-learning).
In our approach, we train a hyper-network with constrained optimization to
modify a fact without affecting the rest of the knowledge; the trained
hyper-network is then used to predict the weight update at test time. We show
KnowledgeEditor's efficacy with two popular architectures and
knowledge-intensive tasks: i) a BERT model fine-tuned for fact-checking, and
ii) a sequence-to-sequence BART model for question answering. With our method,
changing a prediction on the specific wording of a query tends to result in a
consistent change in predictions also for its paraphrases. We show that this
can be further encouraged by exploiting (e.g., automatically-generated)
paraphrases during training. Interestingly, our hyper-network can be regarded
as a 'probe' revealing which components need to be changed to manipulate
factual knowledge; our analysis shows that the updates tend to be concentrated
on a small subset of components. Source code available at
this https URL

本文介绍 KnowledgeEditor 一种方法，用于编辑语言模型中的知识，以修复 “错误” 或意外预测，且不需要重新训练或以元学习为前提条件。作者使用受约束的优化来训练一个超网络，该超网络可预测权重更新。作者展示了 KnowledgeEditor 的有效性，其支持两种流行的体系结构和知识密集型任务：一种是针对事实核对进行细微调整的 BERT 模型，另一种是适用于问答的序列到序列 BART 模型。