We propose a novel text-to-speech (TTS) framework centered around a neural
transducer. Our approach divides the whole TTS pipeline into semantic-level
sequence-to-sequence (seq2seq) modeling and fine-grained acoustic modeling
stages, utilizing discrete semantic tokens obtained from wav2vec2.0 embeddings.
For a robust and efficient alignment modeling, we employ a neural transducer
named token transducer for the semantic token prediction, benefiting from its
hard monotonic alignment constraints. Subsequently, a non-autoregressive (NAR)
speech generator efficiently synthesizes waveforms from these semantic tokens.
Additionally, a reference speech controls temporal dynamics and acoustic
conditions at each stage. This decoupled framework reduces the training
complexity of TTS while allowing each stage to focus on semantic and acoustic
modeling. Our experimental results on zero-shot adaptive TTS demonstrate that
our model surpasses the baseline in terms of speech quality and speaker
similarity, both objectively and subjectively. We also delve into the inference
speed and prosody control capabilities of our approach, highlighting the
potential of neural transducers in TTS frameworks.

我们提出了一种以神经推导器为中心的新型文本转语音（TTS）框架，将整个 TTS 流程分为语义级序列到序列建模和细粒度声学建模阶段，并利用从 wav2vec2.0 嵌入中获取的离散语义标记。我们采用了一种名为令牌推导器的神经推导器，用于语义标记预测以实现稳健高效的对齐建模，从其固有的硬单调对齐约束中受益。接下来，一个非自回归（NAR）语音生成器从这些语义标记有效地合成波形。此外，参考语音在每个阶段控制时间动态和声学条件。这种解耦的框架降低了 TTS 的训练复杂性，同时使每个阶段能够专注于语义和声学建模。我们在零 - shot 自适应 TTS 上的实验证明，我们的模型在语音质量和说话人相似性方面超过了基准模型，无论是客观上还是主观上。我们还深入探讨了我们方法在推理速度和韵律控制能力方面的优势，突出了神经推导器在 TTS 框架中的潜力。