We introduce VoiceCraft, a token infilling neural codec language model, that
achieves state-of-the-art performance on both speech editing and zero-shot
text-to-speech (TTS) on audiobooks, internet videos, and podcasts. VoiceCraft
employs a Transformer decoder architecture and introduces a token rearrangement
procedure that combines causal masking and delayed stacking to enable
generation within an existing sequence. On speech editing tasks, VoiceCraft
produces edited speech that is nearly indistinguishable from unedited
recordings in terms of naturalness, as evaluated by humans; for zero-shot TTS,
our model outperforms prior SotA models including VALLE and the popular
commercial model XTTS-v2. Crucially, the models are evaluated on challenging
and realistic datasets, that consist of diverse accents, speaking styles,
recording conditions, and background noise and music, and our model performs
consistently well compared to other models and real recordings. In particular,
for speech editing evaluation, we introduce a high quality, challenging, and
realistic dataset named RealEdit. We encourage readers to listen to the demos
at this https URL

VoiceCraft 是一种令牌填充的神经编解码语言模型，不仅在语音编辑和零样本文本到语音的任务上取得了最先进的性能，而且在人工评估中，VoiceCraft 生成的编辑后语音与未编辑的语音几乎无法区分；我们的模型在包含不同口音、说话风格、录音条件、背景噪音和音乐的具有挑战性和真实性的数据集上表现出色，在语音编辑方面，我们引入了一个高质量、具有挑战性和真实性的数据集 RealEdit 作为评估。