While recent advancements in speech language models have achieved significant
progress, they face remarkable challenges in modeling the long acoustic
sequences of neural audio codecs. In this paper, we introduce
\textbf{G}enerative \textbf{P}re-trained \textbf{S}peech \textbf{T}ransformer
(GPST), a hierarchical transformer designed for efficient speech language
modeling. GPST quantizes audio waveforms into two distinct types of discrete
speech representations and integrates them within a hierarchical transformer
architecture, allowing for a unified one-stage generation process and enhancing
Hi-Res audio generation capabilities. By training on large corpora of speeches
in an end-to-end unsupervised manner, GPST can generate syntactically
consistent speech with diverse speaker identities. Given a brief 3-second
prompt, GPST can produce natural and coherent personalized speech,
demonstrating in-context learning abilities. Moreover, our approach can be
easily extended to spoken cross-lingual speech generation by incorporating
multi-lingual semantic tokens and universal acoustic tokens. Experimental
results indicate that GPST significantly outperforms the existing speech
language models in terms of word error rate, speech quality, and speaker
similarity. See https://youngsheen.github.io/GPST/demo for demo samples.

我们介绍了一种用于有效语音语言建模的层次变压器的生成预训练语音变压器 (GPST)，它将音频波形量化为两种不同类型的离散语音表示，并在层次变压器架构中进行集成，从而实现统一的单阶段生成过程并增强 Hi-Res 音频生成能力。通过以端到端无监督的方式在大型语音语料库上进行训练，GPST 可以生成具有多样说话人身份的句法一致的语音。给定一个简短的 3 秒提示，GPST 可以产生自然连贯的个性化语音，展示了上下文学习能力。此外，我们的方法可以通过结合多语义令牌和通用声学令牌轻松扩展到口语跨语言语音生成。实验结果表明，GPST 在词错误率、语音质量和说话人相似度方面明显优于现有语音语言模型。详情请参见 https://youngsheen.github.io/GPST/demo。