In this report, we explore the potential for text diffusion to replace
autoregressive (AR) decoding for the training and deployment of large language
models (LLMs). We are particularly interested to see whether pretrained AR
models can be transformed into text diffusion models through a lightweight
adaptation procedure we call ``AR2Diff''. We begin by establishing a strong
baseline setup for training text diffusion models. Comparing across multiple
architectures and pretraining objectives, we find that training a decoder-only
model with a prefix LM objective is best or near-best across several tasks.
Building on this finding, we test various transfer learning setups for text
diffusion models. On machine translation, we find that text diffusion
underperforms the standard AR approach. However, on code synthesis and
extractive QA, we find diffusion models trained from scratch outperform AR
models in many cases. We also observe quality gains from AR2Diff -- adapting AR
models to use diffusion decoding. These results are promising given that text
diffusion is relatively underexplored and can be significantly faster than AR
decoding for long text generation.

我们研究了文本扩散能否替代自回归（AR）解码用于大型语言模型（LLMs）的训练和部署，并测试了通过一种轻量级适应过程称为 ``AR2Diff'' 将预训练的 AR 模型转换为文本扩散模型的潜力。我们发现，训练仅具有前缀 LM 目标的解码器模型在多项任务中表现最好。此外，我们还观察到通过 AR2Diff 对 AR 模型进行改进，并发现使用扩散解码的扩散模型比 AR 模型在多种情况下表现出更好的结果。这些结果非常有前景，因为相对未被充分探索的文本扩散比长文本生成的 AR 解码速度更快。