We present DiffusionBERT, a new generative masked language model based on
discrete diffusion models. Diffusion models and many pre-trained language
models have a shared training objective, i.e., denoising, making it possible to
combine the two powerful models and enjoy the best of both worlds. On the one
hand, diffusion models offer a promising training strategy that helps improve
the generation quality. On the other hand, pre-trained denoising language
models (e.g., BERT) can be used as a good initialization that accelerates
convergence. We explore training BERT to learn the reverse process of a
discrete diffusion process with an absorbing state and elucidate several
designs to improve it. First, we propose a new noise schedule for the forward
diffusion process that controls the degree of noise added at each step based on
the information of each token. Second, we investigate several designs of
incorporating the time step into BERT. Experiments on unconditional text
generation demonstrate that DiffusionBERT achieves significant improvement over
existing diffusion models for text (e.g., D3PM and Diffusion-LM) and previous
generative masked language models in terms of perplexity and BLEU score.

本文介绍了 DiffusionBERT，一种基于离散扩散模型的新型生成遮蔽语言模型，探讨了通过结合扩散模型和预训练去噪语言模型的能力，进一步提高文本生成质量。实验证明，DiffusionBERT 在文本生成方面的表现明显优于现有的扩散模型和先前的生成遮蔽语言模型。