Despite their groundbreaking performance for many generative modeling tasks,
diffusion models have fallen short on discrete data domains such as natural
language. Crucially, standard diffusion models rely on the well-established
theory of score matching, but efforts to generalize this to discrete structures
have not yielded the same empirical gains. In this work, we bridge this gap by
proposing score entropy, a novel discrete score matching loss that is more
stable than existing methods, forms an ELBO for maximum likelihood training,
and can be efficiently optimized with a denoising variant. We scale our Score
Entropy Discrete Diffusion models (SEDD) to the experimental setting of GPT-2,
achieving highly competitive likelihoods while also introducing distinct
algorithmic advantages. In particular, when comparing similarly sized SEDD and
GPT-2 models, SEDD attains comparable perplexities (normally within $+10\%$ of
and sometimes outperforming the baseline). Furthermore, SEDD models learn a
more faithful sequence distribution (around $4\times$ better compared to GPT-2
models with ancestral sampling as measured by large models), can trade off
compute for generation quality (needing only $16\times$ fewer network
evaluations to match GPT-2), and enables arbitrary infilling beyond the
standard left to right prompting.

使用评分熵离散化损失函数来优化基于扩散模型的生成模型，在与 GPT-2 进行比较时取得了竞争性的似然度，并且在算法上也带来了明显的优势。