Large language models (LLMs) have become the foundation of many applications,
leveraging their extensive capabilities in processing and understanding natural
language. While many open-source LLMs have been released with technical
reports, the lack of training details hinders further research and development.
This paper presents the development of YuLan, a series of open-source LLMs with
$12$ billion parameters. The base model of YuLan is pre-trained on
approximately $1.7$T tokens derived from a diverse corpus, including massive
English, Chinese, and multilingual texts. We design a three-stage pre-training
method to enhance YuLan's overall capabilities. Subsequent phases of training
incorporate instruction-tuning and human alignment, employing a substantial
volume of high-quality synthesized data. To facilitate the learning of complex
and long-tail knowledge, we devise a curriculum-learning framework throughout
across these stages, which helps LLMs learn knowledge in an easy-to-hard
manner. YuLan's training is finished on Jan, 2024 and has achieved performance
on par with state-of-the-art LLMs across various English and Chinese
benchmarks. This paper outlines a comprehensive technical roadmap for
developing LLMs from scratch. Our model and codes are available at
this https URL

该论文介绍了 YuLan 的开发，这是一系列具有 120 亿参数的开源 LLMs，其基础模型在多样化语料库中进行了预训练，并采用了三阶段预训练方法以提高整体能力。通过使用大量高质量合成数据，结合指导调整和人工对齐的后续训练阶段，以及跨这些阶段的课程学习框架来促进复杂和长尾知识的学习，YuLan 已在各种英文和中文基准测试中达到与最先进 LLMs 相媲美的性能。