In this paper, we present ControlSpeech, a text-to-speech (TTS) system
capable of fully cloning the speaker's voice and enabling arbitrary control and
adjustment of speaking style, merely based on a few seconds of audio prompt and
a simple textual style description prompt. Prior zero-shot TTS models and
controllable TTS models either could only mimic the speaker's voice without
further control and adjustment capabilities or were unrelated to
speaker-specific voice generation. Therefore, ControlSpeech focuses on a more
challenging new task-a TTS system with controllable timbre, content, and style
at the same time. ControlSpeech takes speech prompts, content prompts, and
style prompts as inputs and utilizes bidirectional attention and mask-based
parallel decoding to capture corresponding codec representations in a discrete
decoupling codec space. Moreover, we discovered the issue of text style
controllability in a many-to-many mapping fashion and proposed the Style
Mixture Semantic Density (SMSD) model to resolve this problem. SMSD module
which is based on Gaussian mixture density networks, is designed to enhance the
fine-grained partitioning and sampling capabilities of style semantic
information and generate speech with more diverse styles. In terms of
experiments, we make available a controllable model toolkit called
ControlToolkit with a new style controllable dataset, some replicated baseline
models and propose new metrics to evaluate both the control capability and the
quality of generated audio in ControlSpeech. The relevant ablation studies
validate the necessity of each component in ControlSpeech is necessary. We hope
that ControlSpeech can establish the next foundation paradigm of controllable
speech synthesis. The relevant code and demo are available at
this https URL .

控制语音 (ControlSpeech) 是一个文本到语音系统，能够完全克隆说话者的声音，并且可以根据几秒钟的音频提示和简单的文本风格描述提示进行任意控制和调整。