Lyrics often convey information about the songs that are beyond the auditory
dimension, enriching the semantic meaning of movements and musical themes. Such
insights are important in the dance choreography domain. However, most existing
dance synthesis methods mainly focus on music-to-dance generation, without
considering the semantic information. To complement it, we introduce JustLMD, a
new multimodal dataset of 3D dance motion with music and lyrics. To the best of
our knowledge, this is the first dataset with triplet information including
dance motion, music, and lyrics. Additionally, we showcase a cross-modal
diffusion-based network designed to generate 3D dance motion conditioned on
music and lyrics. The proposed JustLMD dataset encompasses 4.6 hours of 3D
dance motion in 1867 sequences, accompanied by musical tracks and their
corresponding English lyrics.

介绍了一个新的多模态数据集 JustLMD，其中包括舞蹈动作、音乐和歌词的三元信息，并展示了一种基于跨模态扩散的网络，用于根据音乐和歌词生成 3D 舞蹈动作。

音乐和歌词驱动的舞蹈合成

Music- and Lyrics-driven Dance Synthesis

Dance choreography for a piece of music is a challenging task, having to be
creative in presenting distinctive stylistic dance elements while taking into
account the musical theme and rhythm. It has been tackled by different
approaches such as similarity retrieval, sequence-to-sequence modeling and
generative adversarial networks, but their generated dance sequences are often
short of motion realism, diversity and music consistency. In this paper, we
propose a Music-to-Dance with Optimal Transport Network (MDOT-Net) for learning
to generate 3D dance choreographies from music. We introduce an optimal
transport distance for evaluating the authenticity of the generated dance
distribution and a Gromov-Wasserstein distance to measure the correspondence
between the dance distribution and the input music. This gives a well defined
and non-divergent training objective that mitigates the limitation of standard
GAN training which is frequently plagued with instability and divergent
generator loss issues. Extensive experiments demonstrate that our MDOT-Net can
synthesize realistic and diverse dances which achieve an organic unity with the
input music, reflecting the shared intentionality and matching the rhythmic
articulation. Sample results are found at
this https URL

本文介绍了一种名为 MDOT-Net 的音乐到舞蹈优化传输网络，用于从音乐中生成真实生动的 3D 舞蹈编排。