Masked (or absorbing) diffusion is actively explored as an alternative to
autoregressive models for generative modeling of discrete data. However,
existing work in this area has been hindered by unnecessarily complex model
formulations and unclear relationships between different perspectives, leading
to suboptimal parameterization, training objectives, and ad hoc adjustments to
counteract these issues. In this work, we aim to provide a simple and general
framework that unlocks the full potential of masked diffusion models. We show
that the continuous-time variational objective of masked diffusion models is a
simple weighted integral of cross-entropy losses. Our framework also enables
training generalized masked diffusion models with state-dependent masking
schedules. When evaluated by perplexity, our models trained on OpenWebText
surpass prior diffusion language models at GPT-2 scale and demonstrate superior
performance on 4 out of 5 zero-shot language modeling tasks. Furthermore, our
models vastly outperform previous discrete diffusion models on pixel-level
image modeling, achieving 2.78~(CIFAR-10) and 3.42 (ImageNet 64$\times$64) bits
per dimension that are comparable or better than autoregressive models of
similar sizes.

掩蔽扩散模型是生成离散数据的自回归模型的替代选择，本论文提出了一个简单且通用的框架，解锁了掩蔽扩散模型的全部潜力，并在 OpenWebText 数据集上训练的模型在困惑度上超过了 GPT-2 模型，并在 5 个零 - shot 语言建模任务中展现出卓越性能，在像素级图像建模中也超过了之前的离散扩散模型。