While Diffusion Generative Models have achieved great success on image
generation tasks, how to efficiently and effectively incorporate them into
speech generation especially translation tasks remains a non-trivial problem.
Specifically, due to the low information density of speech data, the
transformed discrete speech unit sequence is much longer than the corresponding
text transcription, posing significant challenges to existing auto-regressive
models. Furthermore, it is not optimal to brutally apply discrete diffusion on
the speech unit sequence while disregarding the continuous space structure,
which will degrade the generation performance significantly. In this paper, we
propose a novel diffusion model by applying the diffusion forward process in
the \textit{continuous} speech representation space, while employing the
diffusion backward process in the \textit{discrete} speech unit space. In this
way, we preserve the semantic structure of the continuous speech representation
space in the diffusion process and integrate the continuous and discrete
diffusion models. We conduct extensive experiments on the textless direct
speech-to-speech translation task, where the proposed method achieves
comparable results to the computationally intensive auto-regressive baselines
(500 steps on average) with significantly fewer decoding steps (50 steps).

通过在连续语音表示空间中应用向前扩散，同时在离散语音单位空间中应用向后扩散，我们提出了一种新的扩散模型，以此在扩散过程中保留了连续语音表示空间的语义结构，并整合了连续和离散扩散模型。在无文本直接语音翻译任务上进行了大量实验，该方法与计算密集型的自回归基线方法（平均 500 步骤）相比，具有显著较少的解码步骤（50 步骤），并获得了可比较的结果。