As large language models (LLMs) are widely deployed, targeted editing of
their knowledge has become a critical challenge. Recently, advancements in
model editing techniques, such as Rank-One Model Editing (ROME), have paved the
way for updating LLMs with new knowledge. However, the efficacy of these
methods varies across different types of knowledge. This study investigates the
capability of knowledge editing methods to incorporate new knowledge with
varying degrees of "perplexingness", a term we use to describe the initial
difficulty LLMs have in understanding new concepts. We begin by quantifying the
"perplexingness" of target knowledge using pre-edit conditional probabilities,
and assess the efficacy of edits through post-edit conditional probabilities.
Utilizing the widely-used CounterFact dataset, we find significant negative
correlations between the "perplexingness" of the new knowledge and the edit
efficacy across all 12 scenarios. To dive deeper into this phenomenon, we
introduce a novel dataset, HierarchyData, consisting of 99 hyponym-hypernym
pairs across diverse categories. Our analysis reveal that more abstract
concepts (hypernyms) tend to be more perplexing than their specific
counterparts (hyponyms). Further exploration into the influence of knowledge
hierarchy on editing outcomes indicates that knowledge positioned at higher
hierarchical levels is more challenging to modify in some scenarios. Our
research highlights a previously overlooked aspect of LLM editing: the variable
efficacy of editing methods in handling perplexing knowledge. By revealing how
hierarchical relationships can influence editing outcomes, our findings offer
new insights into the challenges of updating LLMs and pave the way for more
nuanced approaches to model editing in the future.

大语言模型知识编辑的挑战在于不同类型的知识以及层次关系对编辑效果的影响。本研究通过量化知识的困惑度并评估编辑方法的效果，发现新知识的困惑度与编辑效果存在负相关。进一步研究发现，抽象概念更困惑，而层次关系对编辑结果有影响。研究提供了更深入的大语言模型编辑挑战以及未来模型编辑的启示。