The field of portrait image animation, driven by speech audio input, has
experienced significant advancements in the generation of realistic and dynamic
portraits. This research delves into the complexities of synchronizing facial
movements and creating visually appealing, temporally consistent animations
within the framework of diffusion-based methodologies. Moving away from
traditional paradigms that rely on parametric models for intermediate facial
representations, our innovative approach embraces the end-to-end diffusion
paradigm and introduces a hierarchical audio-driven visual synthesis module to
enhance the precision of alignment between audio inputs and visual outputs,
encompassing lip, expression, and pose motion. Our proposed network
architecture seamlessly integrates diffusion-based generative models, a
UNet-based denoiser, temporal alignment techniques, and a reference network.
The proposed hierarchical audio-driven visual synthesis offers adaptive control
over expression and pose diversity, enabling more effective personalization
tailored to different identities. Through a comprehensive evaluation that
incorporates both qualitative and quantitative analyses, our approach
demonstrates obvious enhancements in image and video quality, lip
synchronization precision, and motion diversity. Further visualization and
access to the source code can be found at:
this https URL

通过采用扩散法为基础的方法框架，该研究提出了一种音频驱动下的肖像图像动画技术，通过引入层次化的音频驱动视觉合成模块，实现了更准确的音频输入与视觉输出的对齐，包括嘴唇、表情和姿势的动作，并通过定性和定量分析进行综合评估。

Hallo: 分层音频驱动的肖像图像动画综合

Hallo: Hierarchical Audio-Driven Visual Synthesis for Portrait Image  Animation

Millions of images of human faces are captured every single day; but these
photographs portray the likeness of an individual with a fixed pose,
expression, and appearance. Portrait image animation enables the post-capture
adjustment of these attributes from a single image while maintaining a
photorealistic reconstruction of the subject's likeness or identity. Still,
current methods for portrait image animation are typically based on 2D warping
operations or manipulations of a 2D generative adversarial network (GAN) and
lack explicit mechanisms to enforce multi-view consistency. Thus these methods
may significantly alter the identity of the subject, especially when the
viewpoint relative to the camera is changed. In this work, we leverage newly
developed 3D GANs, which allow explicit control over the pose of the image
subject with multi-view consistency. We propose a supervision strategy to
flexibly manipulate expressions with 3D morphable models, and we show that the
proposed method also supports editing appearance attributes, such as age or
hairstyle, by interpolating within the latent space of the GAN. The proposed
technique for portrait image animation outperforms previous methods in terms of
image quality, identity preservation, and pose transfer while also supporting
attribute editing.

本文提出了一种基于新开发的三维生成对抗网络（GANs）的画像图像动画的技术，该技术允许对图像主体的姿态进行明确控制，并展示了该方法在图像质量，身份保护和姿态转移等方面胜过先前的方法，同时支持属性修改。