In this paper, we introduce PanGu-Bot, a Chinese pre-trained open-domain
dialogue generation model based on a large pre-trained language model (PLM)
PANGU-alpha (Zeng et al.,2021). Different from other pre-trained dialogue
models trained over a massive amount of dialogue data from scratch, we aim to
build a powerful dialogue model with relatively fewer data and computation
costs by inheriting valuable language capabilities and knowledge from PLMs. To
this end, we train PanGu-Bot from the large PLM PANGU-alpha, which has been
proven well-performed on a variety of Chinese natural language tasks. We
investigate different aspects of responses generated by PanGu-Bot, including
response quality, knowledge, and safety. We show that PanGu-Bot outperforms
state-of-the-art Chinese dialogue systems (CDIALGPT (Wang et al., 2020), EVA
(Zhou et al., 2021), EVA2.0 (Gu et al., 2022)) w.r.t. the above three aspects.
We also demonstrate that PanGu-Bot can be easily deployed to generate emotional
responses without further training. Throughout our empirical analysis, we also
point out that the PanGu-Bot response quality, knowledge correctness, and
safety are still far from perfect, and further explorations are indispensable
to building reliable and smart dialogue systems. Our model and code will be
available at
this https URL
soon.

本文介绍 PanGu-Bot，一个基于大型预训练语言模型（PLM）PANGU-alpha 的中文预训练开放域对话生成模型。从响应质量、知识和安全三个方面，PanGu-Bot 优于现有的中文对话系统，并且可以轻松地生成情感响应。但是，响应质量、知识正确性和安全性仍然不够完美，需要进一步研究来建立可靠智能的对话系统。

基于预训练语言模型的高效生成式对话预训练技术 - PanGu-Bot

PanGu-Bot: Efficient Generative Dialogue Pre-training from Pre-trained Language Model

Large-scale Pretrained Language Models (PLMs) have become the new paradigm
for Natural Language Processing (NLP). PLMs with hundreds of billions
parameters such as GPT-3 have demonstrated strong performances on natural
language understanding and generation with \textit{few-shot in-context}
learning. In this work, we present our practice on training large-scale
autoregressive language models named PanGu-$\alpha$, with up to 200 billion
parameters. PanGu-$\alpha$ is developed under the MindSpore and trained on a
cluster of 2048 Ascend 910 AI processors. The training parallelism strategy is
implemented based on MindSpore Auto-parallel, which composes five parallelism
dimensions to scale the training task to 2048 processors efficiently, including
data parallelism, op-level model parallelism, pipeline model parallelism,
optimizer model parallelism and rematerialization. To enhance the
generalization ability of PanGu-$\alpha$, we collect 1.1TB high-quality Chinese
data from a wide range of domains to pretrain the model. We empirically test
the generation ability of PanGu-$\alpha$ in various scenarios including text
summarization, question answering, dialogue generation, etc. Moreover, we
investigate the effect of model scales on the few-shot performances across a
broad range of Chinese NLP tasks. The experimental results demonstrate the
superior capabilities of PanGu-$\alpha$ in performing various tasks under
few-shot or zero-shot settings.

本文介绍了在 MindSpore 框架下使用 2048 Ascend 910 AI 处理器集群训练了包含 2000 亿个参数的 PanGu-alpha 预训练语言模型及其并行算法，测试了其在各种场景下的生成能力 (包括文本摘要、问答、对话生成等) 以及探究其在零样本及少样本情况下的表现，证明了其在这些任务中的卓越能力。