Building facial analysis systems that generalize to extreme variations in lighting and facial expressions is a challenging problem that can potentially be alleviated using natural-looking synthetic data. Towards that, we propose LEGAN, a novel synthesis framework that leverages perceptual quality judgments for jointly manipulating lighting and expressions in face images, without requiring paired training data. LEGAN disentangles the lighting and expression subspaces and performs transformations in the feature space before upscaling to the desired output image. The fidelity of the synthetic image is further refined by integrating a perceptual quality estimation model into the LEGAN framework as an auxiliary discriminator. The quality estimation model is learned from face images rendered using multiple synthesis methods and their crowd-sourced naturalness ratings using a margin-based regression loss. Using objective metrics like FID and LPIPS, LEGAN is shown to generate higher quality face images when compared with popular GAN models like pix2pix, CycleGAN and StarGAN for lighting and expression synthesis. We also conduct a perceptual study using images synthesized by LEGAN and other GAN models, trained with and without the quality based auxiliary discriminator, and show the correlation between our quality estimation and visual fidelity. Finally, we demonstrate the effectiveness of LEGAN as training data augmenter for expression recognition and face verification tasks.

提出了LEGAN框架来处理在光照和表情方面极端变化的面部图像合成。该框架使用感知质量判断来处理灯光和表情子空间，并在向量转换到图像时进行提高精度的操作。通过使用客观指标FID和LPIPS以及主观方式来证明LEGAN与StarGAN和StarGAN-v2相比，其生成的面部图像具有更高的质量，并可用于表情识别和面部验证任务的训练数据扩充。

LEGAN：利用人类感知判断解耦方向性光照和面部表情的操纵