This paper presents a new sequence-to-sequence (seq2seq) pre-training method PoDA (Pre-training of Denoising Autoencoders), which learns representations suitable for text generation tasks. Unlike encoder-only (e.g., BERT) or decoder-only (e.g., OpenAI GPT) pre-training approaches, PoDA jointly pre-trains both the encoder and decoder by denoising the noise-corrupted text, and it also has the advantage of keeping the network architecture unchanged in the subsequent fine-tuning stage. Meanwhile, we design a hybrid model of Transformer and pointer-generator networks as the backbone architecture for PoDA. We conduct experiments on two text generation tasks: abstractive summarization, and grammatical error correction. Results on four datasets show that PoDA can improve model performance over strong baselines without using any task-specific techniques and significantly speed up convergence.

本研究提出了一种新的序列到序列（seq2seq）预训练方法PoDA（去噪自编码器的预训练），该方法通过去噪噪声受损文本来联合预训练编码器和解码器，在不使用任何特定任务技术的情况下，提高了模型性能并显著加速了收敛速度，同时，本研究还使用Transformer和指针-生成器网络设计了PoDA的骨干架构，实验表明PoDA可以提高对抽象摘要和语法错误更正这两个文本生成任务的性能。

基于去噪序列到序列预训练的文本生成