We present a unified system to realize one-shot voice conversion (VC) on the
pitch, rhythm, and speaker attributes. Existing works generally ignore the
correlation between prosody and language content, leading to the degradation of
naturalness in converted speech. Additionally, the lack of proper language
features prevents these systems from accurately preserving language content
after conversion. To address these issues, we devise a cascaded modular system
leveraging self-supervised discrete speech units as language representation.
These discrete units provide duration information essential for rhythm
modeling. Our system first extracts utterance-level prosody and speaker
representations from the raw waveform. Given the prosody representation, a
prosody predictor estimates pitch, energy, and duration for each discrete unit
in the utterance. A synthesizer further reconstructs speech based on the
predicted prosody, speaker representation, and discrete units. Experiments show
that our system outperforms previous approaches in naturalness,
intelligibility, speaker transferability, and prosody transferability. Code and
samples are publicly available.

我们提出了一种统一的系统，可以实现一次性的声音转换，包括语调、节奏和说话人属性，并利用自监督离散语音单元作为语言表示来解决语音转换过程中自然度、完整性等问题，并通过实验证明其在自然度、可理解性、说话人迁移性和韵律迁移性方面优于以往的方法。