This technical report presents a diffusion model based framework for face
swapping between two portrait images. The basic framework consists of three
components, i.e., IP-Adapter, ControlNet, and Stable Diffusion's inpainting
pipeline, for face feature encoding, multi-conditional generation, and face
inpainting respectively. Besides, I introduce facial guidance optimization and
CodeFormer based blending to further improve the generation quality.
Specifically, we engage a recent light-weighted customization method (i.e.,
DreamBooth-LoRA), to guarantee the identity consistency by 1) using a rare
identifier "sks" to represent the source identity, and 2) injecting the image
features of source portrait into each cross-attention layer like the text
features. Then I resort to the strong inpainting ability of Stable Diffusion,
and utilize canny image and face detection annotation of the target portrait as
the conditions, to guide ContorlNet's generation and align source portrait with
the target portrait. To further correct face alignment, we add the facial
guidance loss to optimize the text embedding during the sample generation.

基于扩散模型的人脸交换技术，利用 IP-Adapter、ControlNet、以及 Stable Diffusion 的修复工艺实现人脸特征编码、多条件生成和修复操作，更进一步通过面部指导优化和 CodeFormer 进行混合提高生成质量。

基于扩散模型的人脸交换

Face Swap via Diffusion Model

Previous animatable 3D-aware GANs for human generation have primarily focused
on either the human head or full body. However, head-only videos are relatively
uncommon in real life, and full body generation typically does not deal with
facial expression control and still has challenges in generating high-quality
results. Towards applicable video avatars, we present an animatable 3D-aware
GAN that generates portrait images with controllable facial expression, head
pose, and shoulder movements. It is a generative model trained on unstructured
2D image collections without using 3D or video data. For the new task, we base
our method on the generative radiance manifold representation and equip it with
learnable facial and head-shoulder deformations. A dual-camera rendering and
adversarial learning scheme is proposed to improve the quality of the generated
faces, which is critical for portrait images. A pose deformation processing
network is developed to generate plausible deformations for challenging regions
such as long hair. Experiments show that our method, trained on unstructured 2D
images, can generate diverse and high-quality 3D portraits with desired control
over different properties.

基于非结构化的 2D 图像集合，我们提出了一种可调节面部表情、头部姿势和肩膀运动的三维感知生成对抗网络，该网络能够生成多样且高质量的三维头像。

AniPortraitGAN：从 2D 图像集合生成可动态调整的 3D 肖像

AniPortraitGAN: Animatable 3D Portrait Generation from 2D Image  Collections

Generating portrait images by controlling the motions of existing faces is an
important task of great consequence to social media industries. For easy use
and intuitive control, semantically meaningful and fully disentangled
parameters should be used as modifications. However, many existing techniques
do not provide such fine-grained controls or use indirect editing methods i.e.
mimic motions of other individuals. In this paper, a Portrait Image Neural
Renderer (PIRenderer) is proposed to control the face motions with the
parameters of three-dimensional morphable face models (3DMMs). The proposed
model can generate photo-realistic portrait images with accurate movements
according to intuitive modifications. Experiments on both direct and indirect
editing tasks demonstrate the superiority of this model. Meanwhile, we further
extend this model to tackle the audio-driven facial reenactment task by
extracting sequential motions from audio inputs. We show that our model can
generate coherent videos with convincing movements from only a single reference
image and a driving audio stream. Our source code is available at
this https URL

使用三维可变形面部模型参数来控制面部动作，生成逼真的肖像图像并实现音频驱动的面部再现。

PIRenderer：通过语义神经渲染实现可控肖像图像生成

PIRenderer: Controllable Portrait Image Generation via Semantic Neural  Rendering

Editing of portrait images is a very popular and important research topic
with a large variety of applications. For ease of use, control should be
provided via a semantically meaningful parameterization that is akin to
computer animation controls. The vast majority of existing techniques do not
provide such intuitive and fine-grained control, or only enable coarse editing
of a single isolated control parameter. Very recently, high-quality
semantically controlled editing has been demonstrated, however only on
synthetically created StyleGAN images. We present the first approach for
embedding real portrait images in the latent space of StyleGAN, which allows
for intuitive editing of the head pose, facial expression, and scene
illumination in the image. Semantic editing in parameter space is achieved
based on StyleRig, a pretrained neural network that maps the control space of a
3D morphable face model to the latent space of the GAN. We design a novel
hierarchical non-linear optimization problem to obtain the embedding. An
identity preservation energy term allows spatially coherent edits while
maintaining facial integrity. Our approach runs at interactive frame rates and
thus allows the user to explore the space of possible edits. We evaluate our
approach on a wide set of portrait photos, compare it to the current state of
the art, and validate the effectiveness of its components in an ablation study.

本文提出了一种基于 StyleGAN 的新方法，将真实肖像图片嵌入潜空间，从而提供了对肖像图像头部姿态、面部表情和场景照明的直观编辑。通过 StyleRig，即一个预训练的神经网络，来实现参数空间中的语义编辑。我们设计了一种新的分层非线性优化问题来获得嵌入。添加了一项身份保护能量项，以保持面部完整性和空间连贯性。