In this study, we introduce a methodology for human image animation by
leveraging a 3D human parametric model within a latent diffusion framework to
enhance shape alignment and motion guidance in curernt human generative
techniques. The methodology utilizes the SMPL(Skinned Multi-Person Linear)
model as the 3D human parametric model to establish a unified representation of
body shape and pose. This facilitates the accurate capture of intricate human
geometry and motion characteristics from source videos. Specifically, we
incorporate rendered depth images, normal maps, and semantic maps obtained from
SMPL sequences, alongside skeleton-based motion guidance, to enrich the
conditions to the latent diffusion model with comprehensive 3D shape and
detailed pose attributes. A multi-layer motion fusion module, integrating
self-attention mechanisms, is employed to fuse the shape and motion latent
representations in the spatial domain. By representing the 3D human parametric
model as the motion guidance, we can perform parametric shape alignment of the
human body between the reference image and the source video motion.
Experimental evaluations conducted on benchmark datasets demonstrate the
methodology's superior ability to generate high-quality human animations that
accurately capture both pose and shape variations. Furthermore, our approach
also exhibits superior generalization capabilities on the proposed wild
dataset. Project page: this https URL

通过在潜在扩散框架中利用 3D 人体参数模型，介绍了一种人像动画的方法，以增强当前人体生成技术中的形状对齐和运动引导，通过捕捉源视频中的复杂人体几何和运动特征，利用深度图像、法线图和语义图来丰富潜在扩散模型的综合三维形状和详细姿势属性的条件，通过多层运动融合模块在空间域中融合形状和运动潜在表示，并通过以 3D 人体参数模型作为运动引导来执行参变形对齐，实验证明该方法能够生成准确捕捉姿势和形状变化的高质量人体动画，并且在所提出的数据集上具有优越的泛化能力。

Champ: 可控且一致的人物图像动画与三维参数引导

Champ: Controllable and Consistent Human Image Animation with 3D  Parametric Guidance

Text-based diffusion models have exhibited remarkable success in generation
and editing, showing great promise for enhancing visual content with their
generative prior. However, applying these models to video super-resolution
remains challenging due to the high demands for output fidelity and temporal
consistency, which is complicated by the inherent randomness in diffusion
models. Our study introduces Upscale-A-Video, a text-guided latent diffusion
framework for video upscaling. This framework ensures temporal coherence
through two key mechanisms: locally, it integrates temporal layers into U-Net
and VAE-Decoder, maintaining consistency within short sequences; globally,
without training, a flow-guided recurrent latent propagation module is
introduced to enhance overall video stability by propagating and fusing latent
across the entire sequences. Thanks to the diffusion paradigm, our model also
offers greater flexibility by allowing text prompts to guide texture creation
and adjustable noise levels to balance restoration and generation, enabling a
trade-off between fidelity and quality. Extensive experiments show that
Upscale-A-Video surpasses existing methods in both synthetic and real-world
benchmarks, as well as in AI-generated videos, showcasing impressive visual
realism and temporal consistency.

通过文本引导的潜在扩散框架，实现视频升尺度，并在保持时域一致性和质量平衡之间提供更大灵活性。