Recent language model-based text-to-speech (TTS) frameworks demonstrate
scalability and in-context learning capabilities. However, they suffer from
robustness issues due to the accumulation of errors in speech unit predictions
during autoregressive language modeling. In this paper, we propose a phonetic
enhanced language modeling method to improve the performance of TTS models. We
leverage self-supervised representations that are phonetically rich as the
training target for the autoregressive language model. Subsequently, a
non-autoregressive model is employed to predict discrete acoustic codecs that
contain fine-grained acoustic details. The TTS model focuses solely on
linguistic modeling during autoregressive training, thereby reducing the error
propagation that occurs in non-autoregressive training. Both objective and
subjective evaluations validate the effectiveness of our proposed method.

我们提出了一种声学增强的语言建模方法，通过使用语音单元预测中的错误积累来改善 TTS 模型的性能，通过自我监督表示来作为自回归语言模型的训练目标，并采用非自回归模型来预测包含细粒度声学细节的离散声学编解码器，以减少非自回归训练中的错误传播，客观和主观评估验证了我们提出方法的有效性。

语音增强的语言建模用于文本到语音合成

Phonetic Enhanced Language Modeling for Text-to-Speech Synthesis

We present Emu, a Transformer-based multimodal foundation model, which can
seamlessly generate images and texts in multimodal context. This omnivore model
can take in any single-modality or multimodal data input indiscriminately
(e.g., interleaved image, text and video) through a one-model-for-all
autoregressive training process. First, visual signals are encoded into
embeddings, and together with text tokens form an interleaved input sequence.
Emu is then end-to-end trained with a unified objective of classifying the next
text token or regressing the next visual embedding in the multimodal sequence.
This versatile multimodality empowers the exploration of diverse pretraining
data sources at scale, such as videos with interleaved frames and text,
webpages with interleaved images and text, as well as web-scale image-text
pairs and video-text pairs. Emu can serve as a generalist multimodal interface
for both image-to-text and text-to-image tasks, and supports in-context image
and text generation. Across a broad range of zero-shot/few-shot tasks including
image captioning, visual question answering, video question answering and
text-to-image generation, Emu demonstrates superb performance compared to
state-of-the-art large multimodal models. Extended capabilities such as
multimodal assistants via instruction tuning are also demonstrated with
impressive performance.

本文介绍了 Emu—— 一种基于 Transformer 的多模态基础模型，它可以在多模态语境中无缝生成图像和文本。该模型可通过单一模型进行全自回归训练，将任何单模态或多模态数据输入混合 (例如，交错的图像、文本和视频)，并表现得十分出色。