Training data plays a pivotal role in AI models. Large language models (LLMs)
are trained with massive amounts of documents, and their parameters hold
document-related contents. Recently, several studies identified
content-specific locations in LLMs by examining the parameters. Instead of the
post hoc interpretation, we propose another approach. We propose document-wise
memory architecture to track document memories in training. The proposed
architecture maps document representations to memory entries, which softly mask
memories in the forward process of LLMs. Additionally, we propose document
guidance loss, which increases the likelihood of text with document memories
and reduces the likelihood of the text with the memories of other documents.
Experimental results on Wikitext-103-v1 with Pythia-1B show that the proposed
methods provide different memory entries for documents and high recall of
document-related content in generation with trained document-wise memories.

利用大数据训练的 AI 模型在参数中存储着与文档相关的内容。本研究提出了一种基于文档记忆的架构，通过映射文档表示到记忆条目来追踪训练中的文档记忆。同时，引入了文档引导损失，增加带有文档记忆的文本的可能性，减少其他文档记忆的文本的可能性。实验结果表明，所提出的方法在生成文档相关内容时，在记忆条目和高召回率方面具有优势。