In this paper, we propose a singing voice synthesis model, Karaoker-SSL, that
is trained only on text and speech data as a typical multi-speaker acoustic
model. It is a low-resource pipeline that does not utilize any singing data
end-to-end, since its vocoder is also trained on speech data. Karaoker-SSL is
conditioned by self-supervised speech representations in an unsupervised
manner. We preprocess these representations by selecting only a subset of their
task-correlated dimensions. The conditioning module is indirectly guided to
capture style information during training by multi-tasking. This is achieved
with a Conformer-based module, which predicts the pitch from the acoustic
model's output. Thus, Karaoker-SSL allows singing voice synthesis without
reliance on hand-crafted and domain-specific features. There are also no
requirements for text alignments or lyrics timestamps. To refine the voice
quality, we employ a U-Net discriminator that is conditioned on the target
speaker and follows a Diffusion GAN training scheme.

本文提出了一种仅使用文本和语音数据训练的多说话人声音合成模型 Karaoker-SSL，它是一个低资源流程，不需要使用任何歌唱数据，因为它的声码器也是在语音数据上训练的。通过无监督方式，Karaoker-SSL 通过自监督语音表示对其进行调节。为了在训练过程中间接指导条件模块捕捉风格信息，使用了一个基于 Conformer 的模块，该模块从声学模型的输出中预测音高。因此，Karaoker-SSL 允许进行歌声合成而不依赖于手工制作的专门领域特征。此外，它不需要文本对齐或歌词时间戳。为了改善声音质量，我们采用了一个以目标说话人为条件的 U-Net 鉴别器，并采用了 Diffusion GAN 训练方案。