Although there are more than 6,500 languages in the world, the pronunciations
of many phonemes sound similar across the languages. When people learn a
foreign language, their pronunciation often reflects their native language's
characteristics. This motivates us to investigate how the speech synthesis
network learns the pronunciation from datasets from different languages. In
this study, we are interested in analyzing and taking advantage of multilingual
speech synthesis network. First, we train the speech synthesis network
bilingually in English and Korean and analyze how the network learns the
relations of phoneme pronunciation between the languages. Our experimental
result shows that the learned phoneme embedding vectors are located closer if
their pronunciations are similar across the languages. Consequently, the
trained networks can synthesize the English speakers' Korean speech and vice
versa. Using this result, we propose a training framework to utilize
information from a different language. To be specific, we pre-train a speech
synthesis network using datasets from both high-resource language and
low-resource language, then we fine-tune the network using the low-resource
language dataset. Finally, we conducted more simulations on 10 different
languages to show it is generally extendable to other languages.

通过在英语和韩语中训练语音合成网络，本研究分析了网络如何学习不同语言之间的拼音发音关系，并提出了一个基于不同语言信息的训练框架，可以应用于低资源语言。结果表明，该框架可扩展到其他语言，并成功地合成了英语和韩语之间的语音。