The scaling of large language models has greatly improved natural language
understanding, generation, and reasoning. In this work, we develop a system
that trained a trillion-parameter language model on a cluster of Ascend 910 AI
processors and MindSpore framework, and present the language model with 1.085T
parameters named PanGu-{\Sigma}. With parameter inherent from PanGu-{\alpha},
we extend the dense Transformer model to sparse one with Random Routed Experts
(RRE), and efficiently train the model over 329B tokens by using Expert
Computation and Storage Separation(ECSS). This resulted in a 6.3x increase in
training throughput through heterogeneous computing. Our experimental findings
show that PanGu-{\Sigma} provides state-of-the-art performance in zero-shot
learning of various Chinese NLP downstream tasks. Moreover, it demonstrates
strong abilities when fine-tuned in application data of open-domain dialogue,
question answering, machine translation and code generation.

本文介绍了使用 MindSpore 在 Ascend 910 人工智能处理器集群上训练的具有 1.085T 参数的 PanGu-Sigma 语言模型，其使用 Random Routed Experts (RRE) 的密集 Transformer 模型来扩展，通过使用 Expert Computation 和 Storage Separation (ECSS) 在 329B 令牌上高效地训练。实验结果显示，PanGu-Sigma 在各种中文 NLP 下游任务的零 - shot 学习中提供了最先进的性能。此外，当在开放式对话、问答、机器翻译和代码生成的应用数据中进行微调时，PanGu-Sigma 表现出了强大的能力。