In this work, we study the impact of QA fine-tuning data on downstream
factuality. We show that fine-tuning on lesser-known facts that are poorly
stored during pretraining yields significantly worse factuality than
fine-tuning on well-known facts, even when all facts are seen during
pretraining. We prove this phenomenon theoretically, showing that training on
lesser-known facts can lead the model to ignore subject entity names and
instead output a generic plausible response even when the relevant factual
knowledge is encoded in the model. On three question answering benchmarks
(PopQA, Entity Questions, and MMLU) and two language models (Llama-2-7B and
Mistral-7B), we find that (i) finetuning on a completely factual but
lesser-known subset of the data deteriorates downstream factuality (5-10%) and
(ii) finetuning on a subset of better-known examples matches or outperforms
finetuning on the entire dataset. Ultimately, our results shed light on the
interaction between pretrained knowledge and finetuning data and demonstrate
the importance of taking into account how facts are stored in the pretrained
model when fine-tuning for knowledge-intensive tasks.

本文研究了 QA 微调数据对下游事实性的影响，证明了微调在不太为人知的事实上时，即使在预训练期间看到了所有事实，其事实性远远不如在众所周知的事实上进行微调，我们的结果揭示了预训练知识和微调数据之间的相互作用，并在为知识密集型任务进行微调时强调了预训练模型中事实存储的重要性。