In order for large language model (LLM)-based assistants to effectively adapt
to evolving information needs, it must be possible to update their factual
knowledge through continued training on new data. The standard recipe for doing
so involves continued pre-training on new documents followed by
instruction-tuning on question-answer (QA) pairs. However, we find that LLMs
trained with this recipe struggle to answer questions, even though the
perplexity of documents is minimized. We found that QA pairs are generally
straightforward, while documents are more complex, weaving many factual
statements together in an intricate manner. Therefore, we hypothesize that it
is beneficial to expose LLMs to QA pairs before continued pre-training on
documents so that the process of encoding knowledge from complex documents
takes into account how this knowledge is accessed through questions. Based on
this, we propose pre-instruction-tuning (PIT), a method that instruction-tunes
on questions prior to training on documents. This contrasts with standard
instruction-tuning, which learns how to extract knowledge after training on
documents. Extensive experiments and ablation studies demonstrate that PIT
significantly enhances the ability of LLMs to absorb knowledge from new
documents, outperforming standard instruction-tuning by 17.8%.

通过在文档训练之前进行问答配对，提出了一种称为 PIT（pre-instruction-tuning）的方法，能够显著提高大型语言模型从新文档吸收知识的能力，比传统的指令调整方法的性能提高了 17.8%。