In human-centric content generation, the pre-trained text-to-image models
struggle to produce user-wanted portrait images, which retain the identity of
individuals while exhibiting diverse expressions. This paper introduces our
efforts towards personalized face generation. To this end, we propose a novel
multi-modal face generation framework, capable of simultaneous
identity-expression control and more fine-grained expression synthesis. Our
expression control is so sophisticated that it can be specialized by the
fine-grained emotional vocabulary. We devise a novel diffusion model that can
undertake the task of simultaneously face swapping and reenactment. Due to the
entanglement of identity and expression, it's nontrivial to separately and
precisely control them in one framework, thus has not been explored yet. To
overcome this, we propose several innovative designs in the conditional
diffusion model, including balancing identity and expression encoder, improved
midpoint sampling, and explicitly background conditioning. Extensive
experiments have demonstrated the controllability and scalability of the
proposed framework, in comparison with state-of-the-art text-to-image, face
swapping, and face reenactment methods.

本文介绍了一个新颖的多模式人脸生成框架，能够同时进行身份 - 表情控制和更细粒度的表情合成，并通过创新设计在条件扩散模型中实现身份和表情的分别控制。实验证明了该框架在可控性和可扩展性上的优越性。