The task of generating dance from music is crucial, yet current methods,
which mainly produce joint sequences, lead to outputs that lack intuitiveness
and complicate data collection due to the necessity for precise joint
annotations. We introduce a Dance Any Beat Diffusion model, namely DabFusion,
that employs music as a conditional input to directly create dance videos from
still images, utilizing conditional image-to-video generation principles. This
approach pioneers the use of music as a conditioning factor in image-to-video
synthesis. Our method unfolds in two stages: training an auto-encoder to
predict latent optical flow between reference and driving frames, eliminating
the need for joint annotation, and training a U-Net-based diffusion model to
produce these latent optical flows guided by music rhythm encoded by CLAP.
Although capable of producing high-quality dance videos, the baseline model
struggles with rhythm alignment. We enhance the model by adding beat
information, improving synchronization. We introduce a 2D motion-music
alignment score (2D-MM Align) for quantitative assessment. Evaluated on the
AIST++ dataset, our enhanced model shows marked improvements in 2D-MM Align
score and established metrics. Video results can be found on our project page:
this https URL

通过音乐作为条件输入，直接从静态图像中生成舞蹈视频的 Dance Any Beat Diffusion 模型引入了图像到视频生成原则，采用了音乐作为图像到视频生成的条件因素。