The task of generating dance from music is crucial, yet current methods,
which mainly produce joint sequences, lead to outputs that lack intuitiveness
and complicate data collection due to the necessity for precise joint
annotations. We introduce a Dance Any Beat Diffusion model, namely DabFusion,
that employs music as a conditional input to directly create dance videos from
still images, utilizing conditional image-to-video generation principles. This
approach pioneers the use of music as a conditioning factor in image-to-video
synthesis. Our method unfolds in two stages: training an auto-encoder to
predict latent optical flow between reference and driving frames, eliminating
the need for joint annotation, and training a U-Net-based diffusion model to
produce these latent optical flows guided by music rhythm encoded by CLAP.
Although capable of producing high-quality dance videos, the baseline model
struggles with rhythm alignment. We enhance the model by adding beat
information, improving synchronization. We introduce a 2D motion-music
alignment score (2D-MM Align) for quantitative assessment. Evaluated on the
AIST++ dataset, our enhanced model shows marked improvements in 2D-MM Align
score and established metrics. Video results can be found on our project page:
this https URL

通过音乐作为条件输入，直接从静态图像中生成舞蹈视频的 Dance Any Beat Diffusion 模型引入了图像到视频生成原则，采用了音乐作为图像到视频生成的条件因素。

舞蹈视频生成中的节拍与视觉融合

Dance Any Beat: Blending Beats with Visuals in Dance Video Generation

The goal of conditional image-to-video (cI2V) generation is to create a
believable new video by beginning with the condition, i.e., one image and
text.The previous cI2V generation methods conventionally perform in RGB pixel
space, with limitations in modeling motion consistency and visual continuity.
Additionally, the efficiency of generating videos in pixel space is quite
low.In this paper, we propose a novel approach to address these challenges by
disentangling the target RGB pixels into two distinct components: spatial
content and temporal motions. Specifically, we predict temporal motions which
include motion vector and residual based on a 3D-UNet diffusion model. By
explicitly modeling temporal motions and warping them to the starting image, we
improve the temporal consistency of generated videos. This results in a
reduction of spatial redundancy, emphasizing temporal details. Our proposed
method achieves performance improvements by disentangling content and motion,
all without introducing new structural complexities to the model. Extensive
experiments on various datasets confirm our approach's superior performance
over the majority of state-of-the-art methods in both effectiveness and
efficiency.

通过将目标 RGB 像素分解成空间内容和时间运动两个不同的组成部分，我们提出了一种解决传统 RGB 像素空间方法中涉及建模动作一致性和视觉连贯性限制的新方法。通过显式建模时间运动并将其应用于起始图像，我们改进了生成视频的时间一致性，减少了空间冗余，突出了时间细节。大量实验证实了我们的方法相对于大多数最先进的方法在效果和效率上的卓越性能。

解耦内容与运动进行条件图像到视频的生成

Decouple Content and Motion for Conditional Image-to-Video Generation

Conditional image-to-video (cI2V) generation aims to synthesize a new
plausible video starting from an image (e.g., a person's face) and a condition
(e.g., an action class label like smile). The key challenge of the cI2V task
lies in the simultaneous generation of realistic spatial appearance and
temporal dynamics corresponding to the given image and condition. In this
paper, we propose an approach for cI2V using novel latent flow diffusion models
(LFDM) that synthesize an optical flow sequence in the latent space based on
the given condition to warp the given image. Compared to previous
direct-synthesis-based works, our proposed LFDM can better synthesize spatial
details and temporal motion by fully utilizing the spatial content of the given
image and warping it in the latent space according to the generated
temporally-coherent flow. The training of LFDM consists of two separate stages:
(1) an unsupervised learning stage to train a latent flow auto-encoder for
spatial content generation, including a flow predictor to estimate latent flow
between pairs of video frames, and (2) a conditional learning stage to train a
3D-UNet-based diffusion model (DM) for temporal latent flow generation. Unlike
previous DMs operating in pixel space or latent feature space that couples
spatial and temporal information, the DM in our LFDM only needs to learn a
low-dimensional latent flow space for motion generation, thus being more
computationally efficient. We conduct comprehensive experiments on multiple
datasets, where LFDM consistently outperforms prior arts. Furthermore, we show
that LFDM can be easily adapted to new domains by simply finetuning the image
decoder. Our code is available at this https URL

本文提出了一种使用基于新颖潜在流扩散模型（LFDM）的方法，旨在从给定的图像和条件（例如笑容）中合成一个新的合理视频，该模型可以更好地综合空间细节和时间动作。