Large language models (LLMs) with billions of parameters have demonstrated
outstanding performance on various natural language processing tasks. This
report presents OpenBA, an open-sourced 15B bilingual asymmetric seq2seq model,
to contribute an LLM variant to the Chinese-oriented open-source model
community. We enhance OpenBA with effective and efficient techniques as well as
adopt a three-stage training strategy to train the model from scratch. Our
solution can also achieve very competitive performance with only 380B tokens,
which is better than LLaMA-70B on the BELEBELE benchmark, BLOOM-176B on the
MMLU benchmark, GLM-130B on the C-Eval (hard) benchmark. This report provides
the main details to pre-train an analogous model, including pre-training data
processing, Bilingual Flan data collection, the empirical observations that
inspire our model architecture design, training objectives of different stages,
and other enhancement techniques. We have refactored our code to follow the
design principles of the Huggingface Transformers Library, making it more
convenient for developers to use, and released checkpoints of different
training stages at this https URL More details of our project
are available at this https URL

大型语言模型（LLM）在多个自然语言处理任务上展现出卓越性能。本报告介绍了 OpenBA，这是一个开源的 150 亿双语不对称 seq2seq 模型，为中文导向的开源模型社区做出了贡献。我们通过有效且高效的技术增强了 OpenBA，并采用三阶段训练策略从头开始训练模型。我们的解决方案在只有 380B 令牌时也能达到与 LLaMA-70B 在 BELEBELE 基准上、BLOOM-176B 在 MMLU 基准上、GLM-130B 在 C-Eval（难）基准上相媲美的性能。本报告提供了有关类似模型的预训练的主要细节，包括预训练数据处理、双语 Flan 数据收集、启发我们模型架构设计的经验观察、不同阶段的训练目标以及其他增强技术。我们重构了代码以符合 Huggingface Transformers Library 的设计原则，使其对开发者更加方便使用，并在 https URL（此处提供链接）上发布了不同训练阶段的检查点。有关我们项目的更多详细信息，请访问 https URL（此处提供链接）。