Zero-shot voice conversion (VC) aims to transform the timbre of a source speaker into any previously unseen target speaker, while preserving the original linguistic content. Despite notable progress, attaining a degree of speaker similarity and naturalness on par with ground truth recordings continues to pose great challenge. In this paper, we propose CTEFM-VC, a zero-shot VC framework that leverages Content-aware Timbre Ensemble modeling and Flow Matching. Specifically, CTEFM-VC disentangles utterances into linguistic content and timbre representations, subsequently utilizing a conditional flow matching model and a vocoder to reconstruct the mel-spectrogram and waveform. To enhance its timbre modeling capability and the naturalness of generated speech, we propose a context-aware timbre ensemble modeling approach that adaptively integrates diverse speaker verification embeddings and enables the joint utilization of linguistic and timbre features through a cross-attention module. Experiments show that our CTEFM-VC system surpasses state-of-the-art VC methods in both speaker similarity and naturalness by at least 18.5% and 7.0%.

本研究解决了零样本语音转换中实现说话人相似性和自然度的挑战。提出的CTEFM-VC框架通过内容感知音色集成建模和流匹配，优化了音色建模能力，结合了说话人验证嵌入，并利用交叉注意力模块联合处理语言与音色特征。实验表明，CTEFM-VC系统在说话人相似性和自然度上分别超越了现有最先进的语音转换方法18.5%和7.0%。

基于内容感知音色集成建模与流匹配的零样本语音转换