With the help of discrete neural audio codecs, large language models (LLM)
have increasingly been recognized as a promising methodology for zero-shot
Text-to-Speech (TTS) synthesis. However, sampling based decoding strategies
bring astonishing diversity to generation, but also pose robustness issues such
as typos, omissions and repetition. In addition, the high sampling rate of
audio also brings huge computational overhead to the inference process of
autoregression. To address these issues, we propose VALL-E R, a robust and
efficient zero-shot TTS system, building upon the foundation of VALL-E.
Specifically, we introduce a phoneme monotonic alignment strategy to strengthen
the connection between phonemes and acoustic sequence, ensuring a more precise
alignment by constraining the acoustic tokens to match their associated
phonemes. Furthermore, we employ a codec-merging approach to downsample the
discrete codes in shallow quantization layer, thereby accelerating the decoding
speed while preserving the high quality of speech output. Benefiting from these
strategies, VALL-E R obtains controllablity over phonemes and demonstrates its
strong robustness by approaching the WER of ground truth. In addition, it
requires fewer autoregressive steps, with over 60% time reduction during
inference. This research has the potential to be applied to meaningful
projects, including the creation of speech for those affected by aphasia. Audio
samples will be available at: this https URL

利用离散神经音频编解码器，我们提出了 VALL-E R，一个强大且高效的零 - shot 语音合成系统，通过引入音素单调对齐策略和编解码器合并方法，提高了音素的可控性，加快了解码速度，并取得了接近真实语音的字词错误率和超过 60% 的推理时间减少。

VALL-E R：鲁棒高效的零射文本语音合成方法：单调对齐

VALL-E R: Robust and Efficient Zero-Shot Text-to-Speech Synthesis via  Monotonic Alignment

This paper introduces VALL-E 2, the latest advancement in neural codec
language models that marks a milestone in zero-shot text-to-speech synthesis
(TTS), achieving human parity for the first time. Based on its predecessor,
VALL-E, the new iteration introduces two significant enhancements: Repetition
Aware Sampling refines the original nucleus sampling process by accounting for
token repetition in the decoding history. It not only stabilizes the decoding
but also circumvents the infinite loop issue. Grouped Code Modeling organizes
codec codes into groups to effectively shorten the sequence length, which not
only boosts inference speed but also addresses the challenges of long sequence
modeling. Our experiments on the LibriSpeech and VCTK datasets show that VALL-E
2 surpasses previous systems in speech robustness, naturalness, and speaker
similarity. It is the first of its kind to reach human parity on these
benchmarks. Moreover, VALL-E 2 consistently synthesizes high-quality speech,
even for sentences that are traditionally challenging due to their complexity
or repetitive phrases. The advantages of this work could contribute to valuable
endeavors, such as generating speech for individuals with aphasia or people
with amyotrophic lateral sclerosis. Demos of VALL-E 2 will be posted to
this https URL

VALL-E 2 是零样本文本到语音合成领域最新的神经编解码模型，首次实现了人类水平的合成，并引入了两个重要的改进：重复感知采样和编解码编组建模。实验结果表明 VALL-E 2 在语音鲁棒性、自然程度和说话者相似性方面均超过了之前的系统，在 LibriSpeech 和 VCTK 数据集上取得了人类水平的合成效果，具有很大的潜在应用价值。