We propose SelfVC, a training strategy to iteratively improve a voice
conversion model with self-synthesized examples. Previous efforts on voice
conversion focus on explicitly disentangling speech representations to
separately encode speaker characteristics and linguistic content. However,
disentangling speech representations to capture such attributes using
task-specific loss terms can lead to information loss by discarding finer
nuances of the original signal. In this work, instead of explicitly
disentangling attributes with loss terms, we present a framework to train a
controllable voice conversion model on entangled speech representations derived
from self-supervised learning and speaker verification models. First, we
develop techniques to derive prosodic information from the audio signal and SSL
representations to train predictive submodules in the synthesis model. Next, we
propose a training strategy to iteratively improve the synthesis model for
voice conversion, by creating a challenging training objective using
self-synthesized examples. In this training approach, the current state of the
synthesis model is used to generate voice-converted variations of an utterance,
which serve as inputs for the reconstruction task, ensuring a continuous and
purposeful refinement of the model. We demonstrate that incorporating such
self-synthesized examples during training improves the speaker similarity of
generated speech as compared to a baseline voice conversion model trained
solely on heuristically perturbed inputs. SelfVC is trained without any text
and is applicable to a range of tasks such as zero-shot voice conversion,
cross-lingual voice conversion, and controllable speech synthesis with pitch
and pace modifications. SelfVC achieves state-of-the-art results in zero-shot
voice conversion on metrics evaluating naturalness, speaker similarity, and
intelligibility of synthesized audio.

我们提出了 SelfVC，一种训练策略，通过自我合成的示例来迭代改进语音转换模型。该模型在训练过程中使用自我监督学习和说话人验证模型生成的交织的语音表示来训练具有可控性的语音转换模型，并通过创建具有挑战性的自我合成示例来不断改进模型。SelfVC 在没有文本的情况下进行训练，并适用于零样本语音转换、跨语言语音转换以及具有音高和语速修改的可控语音合成任务。在自然度、说话人相似度和可理解性的评估指标上，SelfVC 在零样本语音转换方面取得了最先进的结果。