Recent progress in music generation has been remarkably advanced by the
state-of-the-art MusicLM, which comprises a hierarchy of three LMs,
respectively, for semantic, coarse acoustic, and fine acoustic modelings. Yet,
sampling with the MusicLM requires processing through these LMs one by one to
obtain the fine-grained acoustic tokens, making it computationally expensive
and prohibitive for a real-time generation. Efficient music generation with a
quality on par with MusicLM remains a significant challenge. In this paper, we
present MeLoDy (M for music; L for LM; D for diffusion), an LM-guided diffusion
model that generates music audios of state-of-the-art quality meanwhile
reducing 95.7% or 99.6% forward passes in MusicLM, respectively, for sampling
10s or 30s music. MeLoDy inherits the highest-level LM from MusicLM for
semantic modeling, and applies a novel dual-path diffusion (DPD) model and an
audio VAE-GAN to efficiently decode the conditioning semantic tokens into
waveform. DPD is proposed to simultaneously model the coarse and fine acoustics
by incorporating the semantic information into segments of latents effectively
via cross-attention at each denoising step. Our experimental results suggest
the superiority of MeLoDy, not only in its practical advantages on sampling
speed and infinitely continuable generation, but also in its state-of-the-art
musicality, audio quality, and text correlation.
Our samples are available at this https URL

介绍了 MeLoDy，一种基于 LM 引导扩散模型，可以高效地生成音乐音频，并同时减少音乐 LM 中 95.7% 或 99.6% 的前向处理，实现了可持续且具有良好音质和文本相关性的音乐生成。