The modern autoregressive Large Language Models (LLMs) have achieved
outstanding performance on NLP benchmarks, and they are deployed in the real
world. However, they still suffer from limitations of the autoregressive
training paradigm. For example, autoregressive token generation is notably slow
and can be prone to \textit{exposure bias}. The diffusion-based language models
were proposed as an alternative to autoregressive generation to address some of
these limitations. We evaluate the recently proposed Score Entropy Discrete
Diffusion (SEDD) approach and show it is a promising alternative to
autoregressive generation but it has some short-comings too. We empirically
demonstrate the advantages and challenges of SEDD, and observe that SEDD
generally matches autoregressive models in perplexity and on benchmarks such as
HellaSwag, Arc or WinoGrande. Additionally, we show that in terms of inference
latency, SEDD can be up to 4.5$\times$ more efficient than GPT-2. While SEDD
allows conditioning on tokens at abitrary positions, SEDD appears slightly
weaker than GPT-2 for conditional generation given short prompts. Finally, we
reproduced the main results from the original SEDD paper.

现代自回归型大型语言模型在自然语言处理基准测试中表现出色并应用于实际领域。然而，它们仍然受到自回归训练范式的一些限制。本文评估了最近提出的基于得分熵离散扩散 (SEDD) 的方法，并表明它是自回归生成的有希望的替代方案，但也存在一些不足之处。我们实证地展示了 SEDD 的优势和挑战，并观察到 SEDD 在困惑度和 HellaSwag、Arc 或 WinoGrande 等基准测试上一般与自回归模型相匹配。此外，我们还展示了 SEDD 在推理延迟方面比 GPT-2 高效多达 4.5 倍。虽然 SEDD 允许在任意位置上进行条件生成，但在给定短提示的条件生成方面，SEDD 似乎比 GPT-2 稍逊一筹。最后，我们复现了原始 SEDD 论文的主要结果。

扩散语言建模的承诺、展望与挑战

Promises, Outlooks and Challenges of Diffusion Language Modeling

Despite a growing interest in diffusion-based language models, existing work
has not shown that these models can attain nontrivial likelihoods on standard
language modeling benchmarks. In this work, we take the first steps towards
closing the likelihood gap between autoregressive and diffusion-based language
models, with the goal of building and releasing a diffusion model which
outperforms a small but widely-known autoregressive model. We pursue this goal
through algorithmic improvements, scaling laws, and increased compute. On the
algorithmic front, we introduce several methodological improvements for the
maximum-likelihood training of diffusion language models. We then study scaling
laws for our diffusion models and find compute-optimal training regimes which
differ substantially from autoregressive models. Using our methods and scaling
analysis, we train and release Plaid 1B, a large diffusion language model which
outperforms GPT-2 124M in likelihood on benchmark datasets and generates fluent
samples in unconditional and zero-shot control settings.

该研究基于算法改进和计算资源扩展来提升扩散式语言模型，成功构建了一款表现优于已有模型的大规模扩散语言模型 Plaid 1B。

基于似然的扩散语言模型

Likelihood-Based Diffusion Language Models

Diffusion-based language models (LMs) have been shown to be competent
generative models that are easy to control at inference and are a promising
alternative to autoregressive LMs. While autoregressive LMs have benefited
immensely from scaling and instruction-based learning, existing studies on
diffusion LMs have been conducted on a relatively smaller scale. Starting with
a recently proposed diffusion model SSD-LM, in this work we explore methods to
scale it from 0.4B to 13B parameters, proposing several techniques to improve
its training and inference efficiency. We call the new model SSD-2. We further
show that this model can be easily finetuned to follow instructions. Finally,
leveraging diffusion models' capability at inference-time control, we show that
SSD-2 facilitates novel ensembles with 100x smaller models that can be
customized and deployed by individual users. We find that compared to
autoregressive models, the collaboration between diffusion models is more
effective, leading to higher-quality and more relevant model responses due to
their ability to incorporate bi-directional contexts.

通过研究和提出 Diffusion-based language models 的新方法，我们成功将 SSD-LM 模型从 0.4B 扩展至 13B 参数，同时提高了其训练和推断的效率，并能够根据指令进行微调，与自回归 LM 相比，Diffusion-based language models 在双向上下文方面的合作更加有效，从而提高了模型响应的质量和相关性。