We present a unified system to realize one-shot voice conversion (VC) on the
pitch, rhythm, and speaker attributes. Existing works generally ignore the
correlation between prosody and language content, leading to the degradation of
naturalness in converted speech. Additionally, the lack of proper language
features prevents these systems from accurately preserving language content
after conversion. To address these issues, we devise a cascaded modular system
leveraging self-supervised discrete speech units as language representation.
These discrete units provide duration information essential for rhythm
modeling. Our system first extracts utterance-level prosody and speaker
representations from the raw waveform. Given the prosody representation, a
prosody predictor estimates pitch, energy, and duration for each discrete unit
in the utterance. A synthesizer further reconstructs speech based on the
predicted prosody, speaker representation, and discrete units. Experiments show
that our system outperforms previous approaches in naturalness,
intelligibility, speaker transferability, and prosody transferability. Code and
samples are publicly available.

我们提出了一种统一的系统，可以实现一次性的声音转换，包括语调、节奏和说话人属性，并利用自监督离散语音单元作为语言表示来解决语音转换过程中自然度、完整性等问题，并通过实验证明其在自然度、可理解性、说话人迁移性和韵律迁移性方面优于以往的方法。

一个自监督离散语音单元的统一单次韵律和说话人转换系统

A unified one-shot prosody and speaker conversion system with self-supervised discrete speech units

One-shot voice conversion (VC), which performs conversion across arbitrary
speakers with only a single target-speaker utterance for reference, can be
effectively achieved by speech representation disentanglement. Existing work
generally ignores the correlation between different speech representations
during training, which causes leakage of content information into the speaker
representation and thus degrades VC performance. To alleviate this issue, we
employ vector quantization (VQ) for content encoding and introduce mutual
information (MI) as the correlation metric during training, to achieve proper
disentanglement of content, speaker and pitch representations, by reducing
their inter-dependencies in an unsupervised manner. Experimental results
reflect the superiority of the proposed method in learning effective
disentangled speech representations for retaining source linguistic content and
intonation variations, while capturing target speaker characteristics. In doing
so, the proposed approach achieves higher speech naturalness and speaker
similarity than current state-of-the-art one-shot VC systems. Our code,
pre-trained models and demo are available at
this https URL

本文提出通过使用向量量化 (VQ) 进行内容编码，并在训练期间引入互信息 (MI) 作为相关度度量，从而在无监督的情况下实现内容、说话人和音高表示的适当分离，以提高单次语音转换系统的性能。该方法在保留源语言内容和语调变化的同时，捕捉目标说话人的特征，实现了比现有单次语音转换系统更高的语音自然度和说话人相似度。