Human dance generation (HDG) aims to synthesize realistic videos from images
and sequences of driving poses. Despite great success, existing methods are
limited to generating videos of a single person with specific backgrounds,
while the generalizability for real-world scenarios with multiple persons and
complex backgrounds remains unclear. To systematically measure the
generalizability of HDG models, we introduce a new task, dataset, and
evaluation protocol of compositional human dance generation (cHDG). Evaluating
the state-of-the-art methods on cHDG, we empirically find that they fail to
generalize to real-world scenarios. To tackle the issue, we propose a novel
zero-shot framework, dubbed MultiDance-Zero, that can synthesize videos
consistent with arbitrary multiple persons and background while precisely
following the driving poses. Specifically, in contrast to straightforward DDIM
or null-text inversion, we first present a pose-aware inversion method to
obtain the noisy latent code and initialization text embeddings, which can
accurately reconstruct the composed reference image. Since directly generating
videos from them will lead to severe appearance inconsistency, we propose a
compositional augmentation strategy to generate augmented images and utilize
them to optimize a set of generalizable text embeddings. In addition,
consistency-guided sampling is elaborated to encourage the background and
keypoints of the estimated clean image at each reverse step to be close to
those of the reference image, further improving the temporal consistency of
generated videos. Extensive qualitative and quantitative results demonstrate
the effectiveness and superiority of our approach.

提出了一种名为 MultiDance-Zero 的零样本框架，用于合成符合任意多人和背景的视频，同时准确地遵循动作姿势，通过深入回溯获取最终清晰图像的背景和关键点，从而进一步提高生成视频的时序一致性。