Diffusion models have gained significant attention in the realm of image
generation due to their exceptional performance. Their success has been
recently expanded to text generation via generating all tokens within a
sequence concurrently. However, natural language exhibits a far more pronounced
sequential dependency in comparison to images, and the majority of existing
language models are trained utilizing a left-to-right auto-regressive approach.
To account for the inherent sequential characteristic of natural language, we
introduce Auto-Regressive Diffusion (AR-Diffusion). AR-Diffusion ensures that
the generation of tokens on the right depends on the generated ones on the
left, a mechanism achieved through employing a dynamic number of denoising
steps that vary based on token position. This results in tokens on the left
undergoing fewer denoising steps than those on the right, thereby enabling them
to generate earlier and subsequently influence the generation of tokens on the
right. In a series of experiments on various text generation tasks including
text summarization, machine translation, and common sense generation,
AR-Diffusion clearly demonstrated the superiority over existing diffusion
language models and that it can be $100\times\sim600\times$ faster when
achieving comparable results. Our code will be publicly released.

介绍了一种新的生成文本的方法 - 自回归扩散 (AR-Diffusion)，它通过引入动态去噪步骤来解决自然语言中的顺序依赖问题，在各种文本生成任务中表现优异。