Large language models (LLMs) encapsulate a vast amount of factual information within their pre-trained weights, as evidenced by their ability to answer diverse questions across different domains. However, this knowledge is inherently limited, relying heavily on the characteristics of the training data. Consequently, using external datasets to incorporate new information or refine the capabilities of LLMs on previously seen information poses a significant challenge. In this study, we compare two common approaches: fine-tuning and retrieval-augmented generation (RAG). We evaluate both approaches on a variety of knowledge-intensive tasks across different topics. Our findings reveal that while fine-tuning offers some improvement, RAG consistently outperforms it, both for existing knowledge encountered during training and entirely new knowledge. Moreover, we find that LLMs struggle to learn new factual information through fine-tuning, and that exposing them to numerous variations of the same fact during training could alleviate this problem.

比较fine-tuning和retrieval-augmented generation (RAG)两种方法，发现RAG在不同主题的知识密集型任务中始终优于fine-tuning，并且LLMs在fine-tuning中难以学习新的事实信息，通过在训练过程中暴露它们于相同事实的多个变体可以缓解这个问题。

微调还是检索？LLMs中知识注入的比较