Despite raw driving videos contain richer information on facial expressions
than intermediate representations such as landmarks in the field of portrait
animation, they are seldom the subject of research. This is due to two
challenges inherent in portrait animation driven with raw videos: 1)
significant identity leakage; 2) Irrelevant background and facial details such
as wrinkles degrade performance. To harnesses the power of the raw videos for
vivid portrait animation, we proposed a pioneering conditional diffusion model
named as MegActor. First, we introduced a synthetic data generation framework
for creating videos with consistent motion and expressions but inconsistent IDs
to mitigate the issue of ID leakage. Second, we segmented the foreground and
background of the reference image and employed CLIP to encode the background
details. This encoded information is then integrated into the network via a
text embedding module, thereby ensuring the stability of the background.
Finally, we further style transfer the appearance of the reference image to the
driving video to eliminate the influence of facial details in the driving
videos. Our final model was trained solely on public datasets, achieving
results comparable to commercial models. We hope this will help the open-source
community.The code is available at
this https URL

通过提出一种先驱性的条件扩散模型 MegActor，我们利用原始驾驶视频进行生动的肖像动画。通过引入合成数据生成框架，分割参考图像的前景和背景，以及风格转换，我们解决了与原始视频驱动的肖像动画相关的问题。

MegActor：利用原始视频的能量进行生动的人像动画

MegActor: Harness the Power of Raw Video for Vivid Portrait Animation

Face swapping has gained significant attention for its varied applications.
The majority of previous face swapping approaches have relied on the seesaw
game training scheme, which often leads to the instability of the model
training and results in undesired samples with blended identities due to the
target identity leakage problem. This paper introduces the Shape Agnostic
Masked AutoEncoder (SAMAE) training scheme, a novel self-supervised approach
designed to enhance face swapping model training. Our training scheme addresses
the limitations of traditional training methods by circumventing the
conventional seesaw game and introducing clear ground truth through its
self-reconstruction training regime. It effectively mitigates identity leakage
by masking facial regions of the input images and utilizing learned
disentangled identity and non-identity features. Additionally, we tackle the
shape misalignment problem with new techniques including perforation confusion
and random mesh scaling, and establishes a new state-of-the-art, surpassing
other baseline methods, preserving both identity and non-identity attributes,
without sacrificing on either aspect.

通过引入自重建训练方案，使用 Shape Agnostic Masked AutoEncoder（SAMAE）来解决传统方法的局限性，包括模型训练不稳定性、目标身份泄露问题和形状错位问题，以实现在保留身份和非身份属性的同时，超越其他基准方法的全新最新技术。