Synthesising appropriate choreographies from music remains an open problem.
We introduce MDLT, a novel approach that frames the choreography generation
problem as a translation task. Our method leverages an existing data set to
learn to translate sequences of audio into corresponding dance poses. We
present two variants of MDLT: one utilising the Transformer architecture and
the other employing the Mamba architecture. We train our method on AIST++ and
PhantomDance data sets to teach a robotic arm to dance, but our method can be
applied to a full humanoid robot. Evaluation metrics, including Average Joint
Error and Frechet Inception Distance, consistently demonstrate that, when given
a piece of music, MDLT excels at producing realistic and high-quality
choreography. The code can be found at github.com/meowatthemoon/MDLT.

用 MDLT 框架，将音乐中的舞蹈姿势转化为相应的舞蹈编排，进行舞蹈生成的翻译任务。

使用序列模型进行音乐舞蹈语言翻译

Music to Dance as Language Translation using Sequence Models

Automatic choreography generation is a challenging task because it often
requires an understanding of two abstract concepts - music and dance - which
are realized in the two different modalities, namely audio and video,
respectively. In this paper, we propose a music-driven choreography generation
system using an auto-regressive encoder-decoder network. To this end, we first
collect a set of multimedia clips that include both music and corresponding
dance motion. We then extract the joint coordinates of the dancer from video
and the mel-spectrogram of music from audio, and train our network using
music-choreography pairs as input. Finally, a novel dance motion is generated
at the inference time when only music is given as an input. We performed a user
study for a qualitative evaluation of the proposed method, and the results show
that the proposed model is able to generate musically meaningful and natural
dance movements given an unheard song.

通过使用自回归编码解码网络设计了一种音乐驱动舞蹈编排生成系统，该网络利用音乐和对应的舞蹈运动，运用多媒体片段进行训练，能够在只有音乐输入的情况下生成新的舞蹈运动。经过用户研究，结果表明该方法可以生成富有音乐感和自然的新舞蹈动作。