Self-supervised learning (SSL) representations from massively multilingual
models offer a promising solution for low-resource language speech tasks.
Despite advancements, language adaptation in TTS systems remains an open
problem. This paper explores the language adaptation capability of ZMM-TTS, a
recent SSL-based multilingual TTS system proposed in our previous work. We
conducted experiments on 12 languages using limited data with various
fine-tuning configurations. We demonstrate that the similarity in phonetics
between the pre-training and target languages, as well as the language
category, affects the target language's adaptation performance. Additionally,
we find that the fine-tuning dataset size and number of speakers influence
adaptability. Surprisingly, we also observed that using paired data for
fine-tuning is not always optimal compared to audio-only data. Beyond speech
intelligibility, our analysis covers speaker similarity, language
identification, and predicted MOS.

本文探讨了基于自监督学习 (SSL) 多语言模型的 ZMM-TTS 语言适应能力，研究发现了预训练语言和目标语言的语音学相似性、语言类别、微调数据集大小以及说话人数量对适应性的影响，并且发现相对于配对数据，仅音频数据在微调中并不总是最优选择。同时，研究还包括了说话人相似度、语言识别以及预测的 MOS。