Recent advances in large language models (LLMs) and development of audio
codecs greatly propel the zero-shot TTS. They can synthesize personalized
speech with only a 3-second speech of an unseen speaker as acoustic prompt.
However, they only support short speech prompts and cannot leverage longer
context information, as required in audiobook and conversational TTS scenarios.
In this paper, we introduce a novel audio codec-based TTS model to adapt
context features with multiple enhancements. Inspired by the success of
Qformer, we propose a multi-modal context-enhanced Qformer (MMCE-Qformer) to
utilize additional multi-modal context information. Besides, we adapt a
pretrained LLM to leverage its understanding ability to predict semantic
tokens, and use a SoundStorm to generate acoustic tokens thereby enhancing
audio quality and speaker similarity. The extensive objective and subjective
evaluations show that our proposed method outperforms baselines across various
context TTS scenarios.

通过将 Qformer 应用于多模态上下文信息，结合预训练的大型语言模型和音频编码器，我们提出了一种适应多种上下文 TTS 场景的全新语音合成模型，通过预测语义令牌和生成声学令牌来提高音频质量和说话人相似性。广泛的客观和主观评估结果表明，我们的方法在各种上下文 TTS 场景中优于基准模型。

基于音频编解码器的零样本文本到语音合成的改进：多模态背景和大型语言模型

Improving Audio Codec-based Zero-Shot Text-to-Speech Synthesis with  Multi-Modal Context and Large Language Model

Large language models (LLMs) have significantly advanced audio processing
through audio codecs that convert audio into discrete tokens, enabling the
application of language modelling techniques to audio data. However,
traditional codecs often operate at high bitrates or within narrow domains such
as speech and lack the semantic clues required for efficient language
modelling. Addressing these challenges, we introduce SemantiCodec, a novel
codec designed to compress audio into fewer than a hundred tokens per second
across diverse audio types, including speech, general audio, and music, without
compromising quality. SemantiCodec features a dual-encoder architecture: a
semantic encoder using a self-supervised AudioMAE, discretized using k-means
clustering on extensive audio data, and an acoustic encoder to capture the
remaining details. The semantic and acoustic encoder outputs are used to
reconstruct audio via a diffusion-model-based decoder. SemantiCodec is
presented in three variants with token rates of 25, 50, and 100 per second,
supporting a range of ultra-low bit rates between 0.31 kbps and 1.43 kbps.
Experimental results demonstrate that SemantiCodec significantly outperforms
the state-of-the-art Descript codec on reconstruction quality. Our results also
suggest that SemantiCodec contains significantly richer semantic information
than all evaluated audio codecs, even at significantly lower bitrates. Our code
and demos are available at this https URL

利用语言模型和音频编解码技术，SemantiCodec 是一种能够以非常低的比特率将各种类型的音频压缩为每秒少于一百个标记的新型编解码器，具有更高的重构质量和更丰富的语义信息。