This paper introduces DanceFusion, a novel framework for reconstructing and generating dance movements synchronized to music, utilizing a Spatio-Temporal Skeleton Diffusion Transformer. The framework adeptly handles incomplete and noisy skeletal data common in short-form dance videos on social media platforms like TikTok. DanceFusion incorporates a hierarchical Transformer-based Variational Autoencoder (VAE) integrated with a diffusion model, significantly enhancing motion realism and accuracy. Our approach introduces sophisticated masking techniques and a unique iterative diffusion process that refines the motion sequences, ensuring high fidelity in both motion generation and synchronization with accompanying audio cues. Comprehensive evaluations demonstrate that DanceFusion surpasses existing methods, providing state-of-the-art performance in generating dynamic, realistic, and stylistically diverse dance motions. Potential applications of this framework extend to content creation, virtual reality, and interactive entertainment, promising substantial advancements in automated dance generation. Visit our project page at https://th-mlab.github.io/DanceFusion/.

本研究解决了短视频平台（如TikTok）中常见的缺失和噪声骨架数据对于舞蹈动作生成的挑战。提出的DanceFusion框架采用分层Transformer变分自编码器与扩散模型的结合，显著提高了运动的真实感和精确度，展示了在动作生成及与音频同步方面的优越性能，有望推动内容创作、虚拟现实和互动娱乐等领域的发展。

舞蹈融合：用于音频驱动舞蹈动作重建的时空骨架扩散变换器