Dance typically involves professional choreography with complex movements
that follow a musical rhythm and can also be influenced by lyrical content. The
integration of lyrics in addition to the auditory dimension, enriches the
foundational tone and makes motion generation more amenable to its semantic
meanings. However, existing dance synthesis methods tend to model motions only
conditioned on audio signals. In this work, we make two contributions to bridge
this gap. First, we propose LM2D, a novel probabilistic architecture that
incorporates a multimodal diffusion model with consistency distillation,
designed to create dance conditioned on both music and lyrics in one diffusion
generation step. Second, we introduce the first 3D dance-motion dataset that
encompasses both music and lyrics, obtained with pose estimation technologies.
We evaluate our model against music-only baseline models with objective metrics
and human evaluations, including dancers and choreographers. The results
demonstrate LM2D is able to produce realistic and diverse dance matching both
lyrics and music. A video summary can be accessed at:
this https URL

通过将音乐和歌词作为条件，在一次扩散生成步骤中创建与歌词和音乐相匹配的逼真且多样化的舞蹈，通过使用多模态扩散模型和一致性蒸馏构建了新颖的概率架构 LM2D，并引入涵盖音乐和歌词的第一个 3D 舞蹈动作数据集，该模型的结果经过客观度量和人类评估表明，能够产生与歌词和音乐匹配的逼真且多样化的舞蹈。