We present Dance2Music-GAN (D2M-GAN), a novel adversarial multi-modal
framework that generates complex musical samples conditioned on dance videos.
Our proposed framework takes dance video frames and human body motions as
input, and learns to generate music samples that plausibly accompany the
corresponding input. Unlike most existing conditional music generation works
that generate specific types of mono-instrumental sounds using symbolic audio
representations (e.g., MIDI), and that usually rely on pre-defined musical
synthesizers, in this work we generate dance music in complex styles (e.g.,
pop, breaking, etc.) by employing a Vector Quantized (VQ) audio representation,
and leverage both its generality and high abstraction capacity of its symbolic
and continuous counterparts. By performing an extensive set of experiments on
multiple datasets, and following a comprehensive evaluation protocol, we assess
the generative qualities of our proposal against alternatives. The attained
quantitative results, which measure the music consistency, beats
correspondence, and music diversity, demonstrate the effectiveness of our
proposed method. Last but not least, we curate a challenging dance-music
dataset of in-the-wild TikTok videos, which we use to further demonstrate the
efficacy of our approach in real-world applications -- and which we hope to
serve as a starting point for relevant future research.

D2M-GAN 是一个基于多模态对抗网络的音乐生成框架，能够根据舞蹈视频生成对应的流行音乐等复杂风格，使用向量量化的音频表示，通过对多个数据集的评估，证明了此方法的有效性，并提供了一个使用 TikTok 视频的数据集用于未来相关研究的起点。