Audio language models have recently emerged as a promising approach for
various audio generation tasks, relying on audio tokenizers to encode waveforms
into sequences of discrete symbols. Audio tokenization often poses a necessary
compromise between code bitrate and reconstruction accuracy. When dealing with
low-bitrate audio codes, language models are constrained to process only a
subset of the information embedded in the audio, which in turn restricts their
generative capabilities. To circumvent these issues, we propose encoding audio
as vector sequences in continuous space $\mathbb R^d$ and autoregressively
generating these sequences using a decoder-only diffusion transformer (ARDiT).
Our findings indicate that ARDiT excels in zero-shot text-to-speech and
exhibits performance that compares to or even surpasses that of
state-of-the-art models. High-bitrate continuous speech representation enables
almost flawless reconstruction, allowing our model to achieve nearly perfect
speech editing. Our experiments reveal that employing Integral Kullback-Leibler
(IKL) divergence for distillation at each autoregressive step significantly
boosts the perceived quality of the samples. Simultaneously, it condenses the
iterative sampling process of the diffusion model into a single step.
Furthermore, ARDiT can be trained to predict several continuous vectors in one
step, significantly reducing latency during sampling. Impressively, one of our
models can generate $170$ ms of $24$ kHz speech per evaluation step with
minimal degradation in performance. Audio samples are available at
this http URL .

通过使用连续空间中的向量序列将音频编码为完美重构的高码率连续语音表示，我们提出了一个仅使用解码器的扩散变换器（ARDiT），在零样本语音合成和其他生成任务中表现出与最先进模型相当或优于其的性能。通过在自回归步骤中采用积分 Kullback-Leibler 散度进行蒸馏，我们显著提高了样本的感知质量，并将扩散模型的迭代采样过程压缩为单一步骤。此外，ARDiT 可以训练以一步预测多个连续向量，从而显著减少采样时的延迟。