In this paper, we take the advantage of previous pre-trained models (PTMs) and propose a novel Chinese Pre-trained Unbalanced Transformer (CPT). Different from previous Chinese PTMs, CPT is designed for both natural language understanding (NLU) and natural language generation (NLG) tasks. CPT consists of three parts: a shared encoder, an understanding decoder, and a generation decoder. Two specific decoders with a shared encoder are pre-trained with masked language modeling (MLM) and denoising auto-encoding (DAE) tasks, respectively. With the partially shared architecture and multi-task pre-training, CPT can (1) learn specific knowledge of both NLU or NLG tasks with two decoders and (2) be fine-tuned flexibly that fully exploits the potential of the model. Moreover, the unbalanced Transformer saves the computational and storage cost, which makes CPT competitive and greatly accelerates the inference of text generation. Experimental results on a wide range of Chinese NLU and NLG tasks show the effectiveness of CPT.

本文介绍了一种新型的中文预训练不平衡变压器（CPT），通过使用自然语言理解（NLU）和自然语言生成（NLG）之间的共享知识来提高性能，结合具有共享编码器的两个特定的解码器以及多任务预训练，CPT能够学习与NLU或NLG任务相关的特定知识，并且可以被灵活地微调以充分利用模型的潜力。此外，不平衡变压器可以节省计算和存储成本，使得CPT具有竞争力并大大加速文本生成的推理。

CPT：一个针对汉语语言理解和生成的预训练不平衡Transformer