When large language models are aligned via supervised fine-tuning, they may
encounter new factual information that was not acquired through pre-training.
It is often conjectured that this can teach the model the behavior of
hallucinating factually incorrect responses, as the model is trained to
generate facts that are not grounded in its pre-existing knowledge. In this
work, we study the impact of such exposure to new knowledge on the capability
of the fine-tuned model to utilize its pre-existing knowledge. To this end, we
design a controlled setup, focused on closed-book QA, where we vary the
proportion of the fine-tuning examples that introduce new knowledge. We
demonstrate that large language models struggle to acquire new factual
knowledge through fine-tuning, as fine-tuning examples that introduce new
knowledge are learned significantly slower than those consistent with the
model's knowledge. However, we also find that as the examples with new
knowledge are eventually learned, they linearly increase the model's tendency
to hallucinate. Taken together, our results highlight the risk in introducing
new factual knowledge through fine-tuning, and support the view that large
language models mostly acquire factual knowledge through pre-training, whereas
fine-tuning teaches them to use it more efficiently.

大型语言模型在通过有监督微调对齐时，会遇到并未通过预训练获得的新的事实信息，从而可能教会模型产生虚假的事实错误响应，导致模型训练生成不基于其现有知识的事实。本研究旨在研究这种新知识暴露对经过微调的模型利用其现有知识的影响。我们设计了一个可控制的实验，并集中于封闭式问答，通过在微调示例中引入新知识的比例来变化。我们证明大型语言模型在微调过程中较难获得新的事实知识，因为引入新知识的微调示例要比与模型现有知识相一致的示例学习速度慢得多。然而，我们还发现随着新知识的学习，它们线性增加了模型产生虚假响应的倾向。综上所述，我们的结果强调通过微调引入新的事实知识的风险，并支持大型语言模型主要通过预训练获取事实知识，而微调则教会它们更有效地使用。

对新知识进行细调的 LLMs 是否鼓励产生幻觉？

Does Fine-Tuning LLMs on New Knowledge Encourage Hallucinations?

Recent work has investigated the interesting question using pre-trained
language models (PLMs) as knowledge bases for answering open questions.
However, existing work is limited in using small benchmarks with high
test-train overlaps. We construct a new dataset of closed-book QA using SQuAD,
and investigate the performance of BART. Experiments show that it is
challenging for BART to remember training facts in high precision, and also
challenging to answer closed-book questions even if relevant knowledge is
retained. Some promising directions are found, including decoupling the
knowledge memorizing process and the QA finetune process, forcing the model to
recall relevant knowledge when question answering.

本研究探索了使用预训练语言模型作为知识库回答开放性问题的方法，构建了一个新的 SQuAD 数据集对 BART 进行了评估，并发现对 BART 来说，在高精度记忆训练事实和回答封闭式问题方面存在挑战，需要将知识记忆过程与 QA 微调过程分开，强制模型在回答问题时记住相关知识。