We present the Melody Guided Music Generation (MMGen) model, the first novel approach using melody to guide the music generation that, despite a pretty simple method and extremely limited resources, achieves excellent performance. Specifically, we first align the melody with audio waveforms and their associated descriptions using the multimodal alignment module. Subsequently, we condition the diffusion module on the learned melody representations. This allows MMGen to generate music that matches the style of the provided audio while also producing music that reflects the content of the given text description. To address the scarcity of high-quality data, we construct a multi-modal dataset, MusicSet, which includes melody, text, and audio, and will be made publicly available. We conduct extensive experiments which demonstrate the superiority of the proposed model both in terms of experimental metrics and actual performance quality.

本研究针对现有音乐生成方法缺乏旋律引导的问题，提出了一种新的旋律引导音乐生成模型（MMGen）。该模型通过多模态对齐模块有效对齐旋律与音频波形及描述，研究结果表明，该方法在生成与提供音频风格相匹配且符合文本描述的音乐方面具有显著优势，且构建的MusicSet数据集将促进该领域的进一步研究。

旋律是音乐生成的关键