While the ability of language models to elicit facts has been widely investigated, how they handle temporally changing facts remains underexplored. We discover Temporal Heads, specific attention heads primarily responsible for processing temporal knowledge through circuit analysis. We confirm that these heads are present across multiple models, though their specific locations may vary, and their responses differ depending on the type of knowledge and its corresponding years. Disabling these heads degrades the model's ability to recall time-specific knowledge while maintaining its general capabilities without compromising time-invariant and question-answering performances. Moreover, the heads are activated not only numeric conditions ("In 2004") but also textual aliases ("In the year ..."), indicating that they encode a temporal dimension beyond simple numerical representation. Furthermore, we expand the potential of our findings by demonstrating how temporal knowledge can be edited by adjusting the values of these heads.

本研究解决了语言模型在处理动态时间事实方面的不足，发现了“时态头”这一特定注意力头，主要负责处理时间知识。研究表明，这些时态头遍布多个模型，其响应因知识类型和对应年份而异，禁用时态头会降低模型回忆时间特定知识的能力，但不影响其普通功能。

时间有其存在的意义吗？时态头：语言模型如何回忆时间特定信息