In recent years, large-scale pre-trained speech language models (SLMs) have
demonstrated remarkable advancements in various generative speech modeling
applications, such as text-to-speech synthesis, voice conversion, and speech
enhancement. These applications typically involve mapping text or speech inputs
to pre-trained SLM representations, from which target speech is decoded. This
paper introduces a new approach, SLMGAN, to leverage SLM representations for
discriminative tasks within the generative adversarial network (GAN) framework,
specifically for voice conversion. Building upon StarGANv2-VC, we add our novel
SLM-based WavLM discriminators on top of the mel-based discriminators along
with our newly designed SLM feature matching loss function, resulting in an
unsupervised zero-shot voice conversion system that does not require text
labels during training. Subjective evaluation results show that SLMGAN
outperforms existing state-of-the-art zero-shot voice conversion models in
terms of naturalness and achieves comparable similarity, highlighting the
potential of SLM-based discriminators for related applications.

介绍了一种新的方法 SLMGAN，它利用 SLM（大规模预训练的语音语言模型）在生成对抗网络（GAN）框架中实现鉴别任务，特别是用于语音转换。通过在基于 mel 的鉴别器之上添加基于 SLM 的 WavLM 鉴别器，并结合新设计的 SLM 特征匹配损失函数，实现了一种无监督的零样本语音转换系统，培训过程中不需要文本标签。主观评估结果表明，SLMGAN 在自然度方面优于现有的零样本语音转换模型，并达到了相似性方面的可比较水平，突显了基于 SLM 的鉴别器在相关应用中的潜力。