This paper studies a transferable phoneme embedding framework that aims to
deal with the cross-lingual text-to-speech (TTS) problem under the few-shot
setting. Transfer learning is a common approach when it comes to few-shot
learning since training from scratch on few-shot training data is bound to
overfit. Still, we find that the naive transfer learning approach fails to
adapt to unseen languages under extremely few-shot settings, where less than 8
minutes of data is provided. We deal with the problem by proposing a framework
that consists of a phoneme-based TTS model and a codebook module to project
phonemes from different languages into a learned latent space. Furthermore, by
utilizing phoneme-level averaged self-supervised learned features, we
effectively improve the quality of synthesized speeches. Experiments show that
using 4 utterances, which is about 30 seconds of data, is enough to synthesize
intelligible speech when adapting to an unseen language using our framework.

本文研究一种可转移的音位嵌入框架，旨在解决少样本跨语言语音合成问题。在 8 分钟以下数据提供的极少样本情况下，作者提出了包括基于音位 TTS 模型和码本模块在内的框架。通过利用音位级平均的自我监督学习特征，实现了合成语音质量的有效改进。实验证明，使用 30 秒左右数据就能够合成出可理解的说话声音。