We propose a novel text-to-speech (TTS) framework centered around a neural
transducer. Our approach divides the whole TTS pipeline into semantic-level
sequence-to-sequence (seq2seq) modeling and fine-grained acoustic modeling
stages, utilizing discrete semantic tokens obtained from wav2vec2.0 embeddings.
For a robust and efficient alignment modeling, we employ a neural transducer
named token transducer for the semantic token prediction, benefiting from its
hard monotonic alignment constraints. Subsequently, a non-autoregressive (NAR)
speech generator efficiently synthesizes waveforms from these semantic tokens.
Additionally, a reference speech controls temporal dynamics and acoustic
conditions at each stage. This decoupled framework reduces the training
complexity of TTS while allowing each stage to focus on semantic and acoustic
modeling. Our experimental results on zero-shot adaptive TTS demonstrate that
our model surpasses the baseline in terms of speech quality and speaker
similarity, both objectively and subjectively. We also delve into the inference
speed and prosody control capabilities of our approach, highlighting the
potential of neural transducers in TTS frameworks.

我们提出了一种以神经推导器为中心的新型文本转语音（TTS）框架，将整个 TTS 流程分为语义级序列到序列建模和细粒度声学建模阶段，并利用从 wav2vec2.0 嵌入中获取的离散语义标记。我们采用了一种名为令牌推导器的神经推导器，用于语义标记预测以实现稳健高效的对齐建模，从其固有的硬单调对齐约束中受益。接下来，一个非自回归（NAR）语音生成器从这些语义标记有效地合成波形。此外，参考语音在每个阶段控制时间动态和声学条件。这种解耦的框架降低了 TTS 的训练复杂性，同时使每个阶段能够专注于语义和声学建模。我们在零 - shot 自适应 TTS 上的实验证明，我们的模型在语音质量和说话人相似性方面超过了基准模型，无论是客观上还是主观上。我们还深入探讨了我们方法在推理速度和韵律控制能力方面的优势，突出了神经推导器在 TTS 框架中的潜力。

基于语义标记预测的两阶段文本到语音的神经转换器

Utilizing Neural Transducers for Two-Stage Text-to-Speech via Semantic  Token Prediction

We introduce a text-to-speech(TTS) framework based on a neural transducer. We
use discretized semantic tokens acquired from wav2vec2.0 embeddings, which
makes it easy to adopt a neural transducer for the TTS framework enjoying its
monotonic alignment constraints. The proposed model first generates aligned
semantic tokens using the neural transducer, then synthesizes a speech sample
from the semantic tokens using a non-autoregressive(NAR) speech generator. This
decoupled framework alleviates the training complexity of TTS and allows each
stage to focus on 1) linguistic and alignment modeling and 2) fine-grained
acoustic modeling, respectively. Experimental results on the zero-shot adaptive
TTS show that the proposed model exceeds the baselines in speech quality and
speaker similarity via objective and subjective measures. We also investigate
the inference speed and prosody controllability of our proposed model, showing
the potential of the neural transducer for TTS frameworks.

基于神经传导器的文本到语音 (TTS) 框架，通过使用从 wav2vec2.0 嵌入中获得的离散语义标记，易于采用神经传导器实现 TTS 框架，从而享受其单调对齐约束；该模型首先通过神经传导器生成对齐的语义标记，然后使用非自回归 (NAR) 语音生成器从语义标记合成语音样本，该分离框架减轻了 TTS 的训练复杂性，使得每个阶段能够分别专注于语言和对齐建模以及细粒度声学建模；在零样本自适应 TTS 上的实验结果表明，通过客观和主观指标，该模型在语音质量和说话人相似度方面超过了基准模型；我们还研究了我们提出的模型的推断速度和语调可控性，展示了神经传导器在 TTS 框架中的潜力。