Achieving nuanced and accurate emulation of human voice has been a
longstanding goal in artificial intelligence. Although significant progress has
been made in recent years, the mainstream of speech synthesis models still
relies on supervised speaker modeling and explicit reference utterances.
However, there are many aspects of human voice, such as emotion, intonation,
and speaking style, for which it is hard to obtain accurate labels. In this
paper, we propose VoxGenesis, a novel unsupervised speech synthesis framework
that can discover a latent speaker manifold and meaningful voice editing
directions without supervision. VoxGenesis is conceptually simple. Instead of
mapping speech features to waveforms deterministically, VoxGenesis transforms a
Gaussian distribution into speech distributions conditioned and aligned by
semantic tokens. This forces the model to learn a speaker distribution
disentangled from the semantic content. During the inference, sampling from the
Gaussian distribution enables the creation of novel speakers with distinct
characteristics. More importantly, the exploration of latent space uncovers
human-interpretable directions associated with specific speaker characteristics
such as gender attributes, pitch, tone, and emotion, allowing for voice editing
by manipulating the latent codes along these identified directions. We conduct
extensive experiments to evaluate the proposed VoxGenesis using both subjective
and objective metrics, finding that it produces significantly more diverse and
realistic speakers with distinct characteristics than the previous approaches.
We also show that latent space manipulation produces consistent and
human-identifiable effects that are not detrimental to the speech quality,
which was not possible with previous approaches. Audio samples of VoxGenesis
can be found at: https://bit.ly/VoxGenesis.

提出了一种无监督的语音合成框架 VoxGenesis，通过探索潜在空间，发现与特定说话者特征相关的可解释方向，实现对声音的编辑，并生成更具多样性和真实性的说话者。