We introduce VoiceCraft, a token infilling neural codec language model, that
achieves state-of-the-art performance on both speech editing and zero-shot
text-to-speech (TTS) on audiobooks, internet videos, and podcasts. VoiceCraft
employs a Transformer decoder architecture and introduces a token rearrangement
procedure that combines causal masking and delayed stacking to enable
generation within an existing sequence. On speech editing tasks, VoiceCraft
produces edited speech that is nearly indistinguishable from unedited
recordings in terms of naturalness, as evaluated by humans; for zero-shot TTS,
our model outperforms prior SotA models including VALLE and the popular
commercial model XTTS-v2. Crucially, the models are evaluated on challenging
and realistic datasets, that consist of diverse accents, speaking styles,
recording conditions, and background noise and music, and our model performs
consistently well compared to other models and real recordings. In particular,
for speech editing evaluation, we introduce a high quality, challenging, and
realistic dataset named RealEdit. We encourage readers to listen to the demos
at this https URL

VoiceCraft 是一种令牌填充的神经编解码语言模型，不仅在语音编辑和零样本文本到语音的任务上取得了最先进的性能，而且在人工评估中，VoiceCraft 生成的编辑后语音与未编辑的语音几乎无法区分；我们的模型在包含不同口音、说话风格、录音条件、背景噪音和音乐的具有挑战性和真实性的数据集上表现出色，在语音编辑方面，我们引入了一个高质量、具有挑战性和真实性的数据集 RealEdit 作为评估。

VoiceCraft: 野外零射语音编辑与文本转语音

VoiceCraft: Zero-Shot Speech Editing and Text-to-Speech in the Wild

We propose a streaming non-autoregressive (non-AR) decoding algorithm to
deliberate the hypothesis alignment of a streaming RNN-T model. Our algorithm
facilitates a simple greedy decoding procedure, and at the same time is capable
of producing the decoding result at each frame with limited right context, thus
enjoying both high efficiency and low latency. These advantages are achieved by
converting the offline Align-Refine algorithm to be streaming-compatible, with
a novel transformer decoder architecture that performs local self-attentions
for both text and audio, and a time-aligned cross-attention at each layer.
Furthermore, we perform discriminative training of our model with the minimum
word error rate (MWER) criterion, which has not been done in the non-AR
decoding literature. Experiments on voice search datasets and Librispeech show
that with reasonable right context, our streaming model performs as well as the
offline counterpart, and discriminative training leads to further WER gain when
the first-pass model has small capacity.

提出了一种流式非自回归（non-AR）解码算法，其利用新型变换器解码器架构在每个帧上执行文本和音频的本地自我关注，并在每个层上进行时间对齐的交叉关注，实现了高效率和低延迟，在声音搜索数据集和 Librispeech 上的实验显示，我们的流式模型表现良好，并且在第一遍模型容量较小的情况下进行判别式训练会进一步提高词误率。