Previous work has showcased the intriguing capability of large language models (LLMs) in retrieving facts and processing context knowledge. However, only limited research exists on the layer-wise capability of LLMs to encode knowledge, which challenges our understanding of their internal mechanisms. In this paper, we devote the first attempt to investigate the layer-wise capability of LLMs through probing tasks. We leverage the powerful generative capability of ChatGPT to construct probing datasets, providing diverse and coherent evidence corresponding to various facts. We employ $\mathcal V$-usable information as the validation metric to better reflect the capability in encoding context knowledge across different layers. Our experiments on conflicting and newly acquired knowledge show that LLMs: (1) prefer to encode more context knowledge in the upper layers; (2) primarily encode context knowledge within knowledge-related entity tokens at lower layers while progressively expanding more knowledge within other tokens at upper layers; and (3) gradually forget the earlier context knowledge retained within the intermediate layers when provided with irrelevant evidence. Code is publicly available at https://github.com/Jometeorie/probing_llama.

通过探究任务，我们在本文中首次尝试研究大型语言模型（LLMs）的逐层能力，并利用ChatGPT的生成能力构建了探测数据集，以提供与各种事实相对应的多样且一致的证据，结果表明LLMs在编码上下文知识方面更倾向于将更多知识码在上层，首先将知识与实体标记在较低层编码，然后在上层逐渐增加其他标记中的知识，并在提供无关证据时逐渐忘记中间层保留的较早的上下文知识。

大型语言模型如何编码上下文知识？一项逐层探测研究