Large-scale Pretrained Language Models (PLMs) have become the new paradigm
for Natural Language Processing (NLP). PLMs with hundreds of billions
parameters such as GPT-3 have demonstrated strong performances on natural
language understanding and generation with \textit{few-shot in-context}
learning. In this work, we present our practice on training large-scale
autoregressive language models named PanGu-$\alpha$, with up to 200 billion
parameters. PanGu-$\alpha$ is developed under the MindSpore and trained on a
cluster of 2048 Ascend 910 AI processors. The training parallelism strategy is
implemented based on MindSpore Auto-parallel, which composes five parallelism
dimensions to scale the training task to 2048 processors efficiently, including
data parallelism, op-level model parallelism, pipeline model parallelism,
optimizer model parallelism and rematerialization. To enhance the
generalization ability of PanGu-$\alpha$, we collect 1.1TB high-quality Chinese
data from a wide range of domains to pretrain the model. We empirically test
the generation ability of PanGu-$\alpha$ in various scenarios including text
summarization, question answering, dialogue generation, etc. Moreover, we
investigate the effect of model scales on the few-shot performances across a
broad range of Chinese NLP tasks. The experimental results demonstrate the
superior capabilities of PanGu-$\alpha$ in performing various tasks under
few-shot or zero-shot settings.

本文介绍了在 MindSpore 框架下使用 2048 Ascend 910 AI 处理器集群训练了包含 2000 亿个参数的 PanGu-alpha 预训练语言模型及其并行算法，测试了其在各种场景下的生成能力 (包括文本摘要、问答、对话生成等) 以及探究其在零样本及少样本情况下的表现，证明了其在这些任务中的卓越能力。