The development of large language models leads to the formation of a pre-train-then-align paradigm, in which the model is typically pre-trained on a large text corpus and undergoes a tuning stage to align the model with human preference or downstream tasks. In this work, we investigate the relationship between pre-training and fine-tuning by fine-tuning multiple intermediate pre-trained model checkpoints. Our results on 18 datasets suggest that i) continual pre-training improves the model in a latent way that unveils after fine-tuning; ii) with extra fine-tuning, the datasets that the model does not demonstrate capability gain much more than those that the model performs well during the pre-training stage; iii) although model benefits significantly through supervised fine-tuning, it may forget previously known domain knowledge and the tasks that are not seen during fine-tuning; iv) the model resembles high sensitivity to evaluation prompts after supervised fine-tuning, but this sensitivity can be alleviated by more pre-training.

本研究探讨了大型语言模型预训练和微调之间的关系，填补了该领域的知识空白。通过微调多个中间预训练模型检查点，发现持续预训练以潜在的方式提升模型性能，并且额外的微调对未展示能力的数据集影响显著。此研究的发现表明微调可能导致知识遗忘，但额外的预训练可以缓解模型对评估提示的敏感性。

阿穆罗与夏尔：分析大型语言模型的预训练与微调关系