This technical report presents a diffusion model based framework for face
swapping between two portrait images. The basic framework consists of three
components, i.e., IP-Adapter, ControlNet, and Stable Diffusion's inpainting
pipeline, for face feature encoding, multi-conditional generation, and face
inpainting respectively. Besides, I introduce facial guidance optimization and
CodeFormer based blending to further improve the generation quality.
Specifically, we engage a recent light-weighted customization method (i.e.,
DreamBooth-LoRA), to guarantee the identity consistency by 1) using a rare
identifier "sks" to represent the source identity, and 2) injecting the image
features of source portrait into each cross-attention layer like the text
features. Then I resort to the strong inpainting ability of Stable Diffusion,
and utilize canny image and face detection annotation of the target portrait as
the conditions, to guide ContorlNet's generation and align source portrait with
the target portrait. To further correct face alignment, we add the facial
guidance loss to optimize the text embedding during the sample generation.

基于扩散模型的人脸交换技术，利用 IP-Adapter、ControlNet、以及 Stable Diffusion 的修复工艺实现人脸特征编码、多条件生成和修复操作，更进一步通过面部指导优化和 CodeFormer 进行混合提高生成质量。