Large Language Model (LLM) based text-to-speech (TTS) systems have
demonstrated remarkable capabilities in handling large speech datasets and
generating natural speech for new speakers. However, LLM-based TTS models are
not robust as the generated output can contain repeating words, missing words
and mis-aligned speech (referred to as hallucinations or attention errors),
especially when the text contains multiple occurrences of the same token. We
examine these challenges in an encoder-decoder transformer model and find that
certain cross-attention heads in such models implicitly learn the text and
speech alignment when trained for predicting speech tokens for a given text. To
make the alignment more robust, we propose techniques utilizing CTC loss and
attention priors that encourage monotonic cross-attention over the text tokens.
Our guided attention training technique does not introduce any new learnable
parameters and significantly improves robustness of LLM-based TTS models.

基于大型语言模型（LLM）的文本到语音（TTS）系统在处理大规模语音数据集和为新说话者生成自然语音方面表现出了显著的能力，但是基于 LLM 的 TTS 模型并不稳健，因为生成的输出可能包含重复的词、缺失的词和错位的语音（即幻觉或注意力错误），特别是当文本中包含多次出现相同令牌的情况。我们在一个编码器 - 解码器 Transformer 模型中研究了这些挑战，并发现这种模型中的某些交叉注意力头在针对给定文本预测语音令牌时隐式地学习了文本和语音的对齐。为了使对齐更加稳健，我们提出了利用 CTC 损失和关注先验的技术，以鼓励对文本标记的单调交叉注意力。我们的引导注意力训练技术不引入任何新的可学习参数，并显著改善了基于 LLM 的 TTS 模型的稳健性。