We introduce AudioLM, a framework for high-quality audio generation with
long-term consistency. AudioLM maps the input audio to a sequence of discrete
tokens and casts audio generation as a language modeling task in this
representation space. We show how existing audio tokenizers provide different
trade-offs between reconstruction quality and long-term structure, and we
propose a hybrid tokenization scheme to achieve both objectives. Namely, we
leverage the discretized activations of a masked language model pre-trained on
audio to capture long-term structure and the discrete codes produced by a
neural audio codec to achieve high-quality synthesis. By training on large
corpora of raw audio waveforms, AudioLM learns to generate natural and coherent
continuations given short prompts. When trained on speech, and without any
transcript or annotation, AudioLM generates syntactically and semantically
plausible speech continuations while also maintaining speaker identity and
prosody for unseen speakers. Furthermore, we demonstrate how our approach
extends beyond speech by generating coherent piano music continuations, despite
being trained without any symbolic representation of music.

AudioLM 是一个高质量的音频生成框架，它通过将输入音频映射为离散令牌序列，并在该表示空间中将音频生成视为一种语言建模任务。我们提出了一种混合标记方案，以实现重建质量和长期结构两个目标，并通过大量的音频波形语料库进行了训练，使其可以生成自然，连贯的音频持续时间。不需要文件、笔录或注释，同时对未见过的讲话者也可以维持其语法和语义合理的音频持续时间。此外，我们还展示了如何通过生成连贯的钢琴音乐持续时间，超越了语音。