Speech representation learning has improved both speech understanding and
speech synthesis tasks for single language. However, its ability in
cross-lingual scenarios has not been explored. In this paper, we extend the
pretraining method for cross-lingual multi-speaker speech synthesis tasks,
including cross-lingual multi-speaker voice cloning and cross-lingual
multi-speaker speech editing. We propose a speech-text joint pretraining
framework, where we randomly mask the spectrogram and the phonemes given a
speech example and its transcription. By learning to reconstruct the masked
parts of the input in different languages, our model shows great improvements
over speaker-embedding-based multi-speaker TTS methods. Moreover, our framework
is end-to-end for both the training and the inference without any finetuning
effort. In cross-lingual multi-speaker voice cloning and cross-lingual
multi-speaker speech editing tasks, our experiments show that our model
outperforms speaker-embedding-based multi-speaker TTS methods.

在跨语言场景中进行了语音表示学习的探索，提出了语音文本联合预训练框架，通过学习重构不同语言的输入来进行预训练，取得了在多语言场景下优于基于说话人嵌入的多说话人 TTS 方法的成果。