The modern autoregressive Large Language Models (LLMs) have achieved
outstanding performance on NLP benchmarks, and they are deployed in the real
world. However, they still suffer from limitations of the autoregressive
training paradigm. For example, autoregressive token generation is notably slow
and can be prone to \textit{exposure bias}. The diffusion-based language models
were proposed as an alternative to autoregressive generation to address some of
these limitations. We evaluate the recently proposed Score Entropy Discrete
Diffusion (SEDD) approach and show it is a promising alternative to
autoregressive generation but it has some short-comings too. We empirically
demonstrate the advantages and challenges of SEDD, and observe that SEDD
generally matches autoregressive models in perplexity and on benchmarks such as
HellaSwag, Arc or WinoGrande. Additionally, we show that in terms of inference
latency, SEDD can be up to 4.5$\times$ more efficient than GPT-2. While SEDD
allows conditioning on tokens at abitrary positions, SEDD appears slightly
weaker than GPT-2 for conditional generation given short prompts. Finally, we
reproduced the main results from the original SEDD paper.

现代自回归型大型语言模型在自然语言处理基准测试中表现出色并应用于实际领域。然而，它们仍然受到自回归训练范式的一些限制。本文评估了最近提出的基于得分熵离散扩散 (SEDD) 的方法，并表明它是自回归生成的有希望的替代方案，但也存在一些不足之处。我们实证地展示了 SEDD 的优势和挑战，并观察到 SEDD 在困惑度和 HellaSwag、Arc 或 WinoGrande 等基准测试上一般与自回归模型相匹配。此外，我们还展示了 SEDD 在推理延迟方面比 GPT-2 高效多达 4.5 倍。虽然 SEDD 允许在任意位置上进行条件生成，但在给定短提示的条件生成方面，SEDD 似乎比 GPT-2 稍逊一筹。最后，我们复现了原始 SEDD 论文的主要结果。