Recent advances in large language models (LLMs) and development of audio
codecs greatly propel the zero-shot TTS. They can synthesize personalized
speech with only a 3-second speech of an unseen speaker as acoustic prompt.
However, they only support short speech prompts and cannot leverage longer
context information, as required in audiobook and conversational TTS scenarios.
In this paper, we introduce a novel audio codec-based TTS model to adapt
context features with multiple enhancements. Inspired by the success of
Qformer, we propose a multi-modal context-enhanced Qformer (MMCE-Qformer) to
utilize additional multi-modal context information. Besides, we adapt a
pretrained LLM to leverage its understanding ability to predict semantic
tokens, and use a SoundStorm to generate acoustic tokens thereby enhancing
audio quality and speaker similarity. The extensive objective and subjective
evaluations show that our proposed method outperforms baselines across various
context TTS scenarios.

通过将 Qformer 应用于多模态上下文信息，结合预训练的大型语言模型和音频编码器，我们提出了一种适应多种上下文 TTS 场景的全新语音合成模型，通过预测语义令牌和生成声学令牌来提高音频质量和说话人相似性。广泛的客观和主观评估结果表明，我们的方法在各种上下文 TTS 场景中优于基准模型。

基于音频编解码器的零样本文本到语音合成的改进：多模态背景和大型语言模型

Improving Audio Codec-based Zero-Shot Text-to-Speech Synthesis with  Multi-Modal Context and Large Language Model

Window-based attention has become a popular choice in vision transformers due
to its superior performance, lower computational complexity, and less memory
footprint. However, the design of hand-crafted windows, which is data-agnostic,
constrains the flexibility of transformers to adapt to objects of varying
sizes, shapes, and orientations. To address this issue, we propose a novel
quadrangle attention (QA) method that extends the window-based attention to a
general quadrangle formulation. Our method employs an end-to-end learnable
quadrangle regression module that predicts a transformation matrix to transform
default windows into target quadrangles for token sampling and attention
calculation, enabling the network to model various targets with different
shapes and orientations and capture rich context information. We integrate QA
into plain and hierarchical vision transformers to create a new architecture
named QFormer, which offers minor code modifications and negligible extra
computational cost. Extensive experiments on public benchmarks demonstrate that
QFormer outperforms existing representative vision transformers on various
vision tasks, including classification, object detection, semantic
segmentation, and pose estimation. The code will be made publicly available at
\href{this https URL}{QFormer}.

该研究提出了一种新颖的四边形注意力（QA）方法，将基于窗口的注意力扩展为一般的四边形公式，并将其整合到平面和分层视觉变换器中，提出了一种新的架构 QFormer，可用于分类、目标检测、语义分割和姿态估计等各种视觉任务，且具有较小的代码修改和可忽略的额外计算成本。