Unsupervised multitask pre-training has been the critical method behind the
recent success of language models (LMs). However, supervised multitask learning
still holds significant promise, as scaling it in the post-training stage
trends towards better generalization. In this paper, we explore supervised
multitask pre-training by proposing Instruction Pre-Training, a framework that
scalably augments massive raw corpora with instruction-response pairs to
pre-train LMs. The instruction-response pairs are generated by an efficient
instruction synthesizer built on open-source models. In our experiments, we
synthesize 200M instruction-response pairs covering 40+ task categories to
verify the effectiveness of Instruction Pre-Training. In pre-training from
scratch, Instruction Pre-Training not only consistently enhances pre-trained
base models but also benefits more from further instruction tuning. In
continual pre-training, Instruction Pre-Training enables Llama3-8B to be
comparable to or even outperform Llama3-70B. Our model, code, and data are
available at this https URL

该研究探索了通过提供指令进行预训练的受监督多任务预训练方法，通过使用开源模型构建的高效指令合成器生成指令 - 响应对，验证了指令预训练的有效性，并证明在从头开始的预训练和持续预训练中都能增强预训练模型的性能。