In this work, we propose MagicDance, a diffusion-based model for 2D human
motion and facial expression transfer on challenging human dance videos.
Specifically, we aim to generate human dance videos of any target identity
driven by novel pose sequences while keeping the identity unchanged. To this
end, we propose a two-stage training strategy to disentangle human motions and
appearance (e.g., facial expressions, skin tone and dressing), consisting of
the pretraining of an appearance-control block and fine-tuning of an
appearance-pose-joint-control block over human dance poses of the same dataset.
Our novel design enables robust appearance control with temporally consistent
upper body, facial attributes, and even background. The model also generalizes
well on unseen human identities and complex motion sequences without the need
for any fine-tuning with additional data with diverse human attributes by
leveraging the prior knowledge of image diffusion models. Moreover, the
proposed model is easy to use and can be considered as a plug-in
module/extension to Stable Diffusion. We also demonstrate the model's ability
for zero-shot 2D animation generation, enabling not only the appearance
transfer from one identity to another but also allowing for cartoon-like
stylization given only pose inputs. Extensive experiments demonstrate our
superior performance on the TikTok dataset.

我们提出了一种基于扩散的模型 MagicDance，用于处理具有挑战性的人类舞蹈视频中的 2D 人体动作和面部表情转换。通过预训练外观控制块和微调外观 - 姿势 - 关节点控制块的两阶段训练策略，我们旨在在保持身份不变的同时生成任何目标身份驱动的人类舞蹈视频。