The parametric knowledge memorized by large language models (LLMs) becomes
outdated quickly. In-context editing (ICE) is currently the most effective
method for updating the knowledge of LLMs. Recent advancements involve
enhancing ICE by modifying the decoding strategy, obviating the need for
altering internal model structures or adjusting external prompts. However, this
enhancement operates across the entire sequence generation, encompassing a
plethora of non-critical tokens. In this work, we introduce $\textbf{A}$daptive
$\textbf{T}$oken $\textbf{Bias}$er ($\textbf{ATBias}$), a new decoding
technique designed to enhance ICE. It focuses on the tokens that are mostly
related to knowledge during decoding, biasing their logits by matching key
entities related to new and parametric knowledge. Experimental results show
that ATBias significantly enhances ICE performance, achieving up to a 32.3%
improvement over state-of-the-art ICE methods while incurring only half the
latency. ATBias not only improves the knowledge editing capabilities of ICE but
can also be widely applied to LLMs with negligible cost.

通过引入自适应令牌偏置器（ATBias），提升了上下文编辑（ICE）性能，实现了编辑知识的能力，并在大型语言模型中广泛应用。

自适应令牌偏置器：通过对关键实体进行偏置来进行知识编辑

Adaptive Token Biaser: Knowledge Editing via Biasing Key Entities

The knowledge within large language models (LLMs) may become outdated
quickly. While in-context editing (ICE) is currently the most effective method
for knowledge editing (KE), it is constrained by the black-box modeling of LLMs
and thus lacks interpretability. Our work aims to elucidate the superior
performance of ICE on the KE by analyzing the impacts of in-context new
knowledge on token-wise distributions. We observe that despite a significant
boost in logits of the new knowledge, the performance of is still hindered by
stubborn knowledge. Stubborn knowledge refers to as facts that have gained
excessive confidence during pretraining, making it hard to edit effectively. To
address this issue and further enhance the performance of ICE, we propose a
novel approach termed $\textbf{De}$coding by $\textbf{C}$ontrasting
$\textbf{K}$nowledge (DeCK). DeCK derives the distribution of the next token by
contrasting the logits obtained from the newly edited knowledge guided by ICE
with those from the unedited parametric knowledge. Our experiments consistently
demonstrate that DeCK enhances the confidence of LLMs in edited facts. For
instance, it improves the performance of LLaMA3-8B-instruct on MQuAKE by up to
219%, demonstrating its capability to strengthen ICE in the editing of stubborn
knowledge. Our work paves the way to develop the both effective and accountable
KE methods for LLMs. (The source code is available at:
$\href{this https URL}{\text{this https URL.}}$ )

通过对上下文新知识的影响进行分析，我们观察到虽然新知识的 logits 显著提升，但由于顽固的知识的存在，in-context editing 的性能仍然受到限制。为了解决这个问题并进一步提高 in-context editing 的性能，我们提出了一种新的方法，称为 DeCK，它通过对比由 in-context editing 引导的新编辑知识和未编辑参数化知识获得的 logits 来导出下一个 token 的分布。实验结果一致表明，DeCK 增强了 LLMs 对编辑事实的置信度，为开发 LLMs 的有效和可追溯的知识编辑方法铺平了道路。