There have been various types of pretraining architectures including
autoencoding models (e.g., BERT), autoregressive models (e.g., GPT), and
encoder-decoder models (e.g., T5). However, none of the pretraining frameworks
performs the best for all tasks of three main categories including natural
language understanding (NLU), unconditional generation, and conditional
generation. We propose a General Language Model (GLM) based on autoregressive
blank infilling to address this challenge. GLM improves blank filling
pretraining by adding 2D positional encodings and allowing an arbitrary order
to predict spans, which results in performance gains over BERT and T5 on NLU
tasks. Meanwhile, GLM can be pretrained for different types of tasks by varying
the number and lengths of blanks. On a wide range of tasks across NLU,
conditional and unconditional generation, GLM outperforms BERT, T5, and GPT
given the same model sizes and data, and achieves the best performance from a
single pretrained model with 1.25x parameters of BERT Large , demonstrating its
generalizability to different downstream tasks.

本文提出了一种基于自回归的空白填充预训练方法称为 GLM，该方法通过添加 2D 位置编码和允许任意顺序来预测跨度来改进填空预训练。实验证明 GLM 在自然语言理解、有条件和无条件生成任务上的性能表现要好于 BERT、T5 和 GPT，预训练时还可以将空白的数量和长度改变以适应不同的任务类型。