Advances in face swapping have enabled the automatic generation of highly
realistic faces. Yet face swaps are perceived differently than when looking at
real faces, with key differences in viewer behavior surrounding the eyes. Face
swapping algorithms generally place no emphasis on the eyes, relying on pixel
or feature matching losses that consider the entire face to guide the training
process. We further investigate viewer perception of face swaps, focusing our
analysis on the presence of an uncanny valley effect. We additionally propose a
novel loss equation for the training of face swapping models, leveraging a
pretrained gaze estimation network to directly improve representation of the
eyes. We confirm that viewed face swaps do elicit uncanny responses from
viewers. Our proposed improvements significant reduce viewing angle errors
between face swaps and their source material. Our method additionally reduces
the prevalence of the eyes as a deciding factor when viewers perform deepfake
detection tasks. Our findings have implications on face swapping for special
effects, as digital avatars, as privacy mechanisms, and more; negative
responses from users could limit effectiveness in said applications. Our gaze
improvements are a first step towards alleviating negative viewer perceptions
via a targeted approach.

面部交换研究中，我们探讨了观察者对面部交换的感知，特别关注了反常谷效应的存在。我们提出了一种新的训练模型方法，通过利用预先训练的凝视估计网络来直接改善对眼睛的表示。研究结果表明，面部交换会引起观察者的反常反应，但通过我们的方法可以显著减少面部交换与源材料之间的视角误差，并减少眼睛在深假检测任务中的重要性。这些发现对于特效制作、数字化形象、隐私保护等领域具有重要意义，用户的负面反应可能限制这些应用的有效性。通过定向方法改善观察者的感知是一个重要的起点。

通过聚焦视线的损失项缓解人脸交换中的怪异感

Towards mitigating uncann(eye)ness in face swaps via gaze-centric loss  terms

Despite rapid advances in computer graphics, creating high-quality
photo-realistic virtual portraits is prohibitively expensive. Furthermore, the
well-know ''uncanny valley'' effect in rendered portraits has a significant
impact on the user experience, especially when the depiction closely resembles
a human likeness, where any minor artifacts can evoke feelings of eeriness and
repulsiveness. In this paper, we present a novel photo-realistic portrait
generation framework that can effectively mitigate the ''uncanny valley''
effect and improve the overall authenticity of rendered portraits. Our key idea
is to employ transfer learning to learn an identity-consistent mapping from the
latent space of rendered portraits to that of real portraits. During the
inference stage, the input portrait of an avatar can be directly transferred to
a realistic portrait by changing its appearance style while maintaining the
facial identity. To this end, we collect a new dataset, Daz-Rendered-Faces-HQ
(DRFHQ), that is specifically designed for rendering-style portraits. We
leverage this dataset to fine-tune the StyleGAN2 generator, using our carefully
crafted framework, which helps to preserve the geometric and color features
relevant to facial identity. We evaluate our framework using portraits with
diverse gender, age, and race variations. Qualitative and quantitative
evaluations and ablation studies show the advantages of our method compared to
state-of-the-art approaches.

本文提出了一个新的照片逼真人像生成框架，可以有效减轻「奇异山谷」效应，提高渲染人像的真实性。使用转移学习从渲染人像的潜在空间学习到真实人像的潜在空间的一致映射，使虚拟角色的人像在外观风格改变的同时保持面部身份的不变，通过细化 StyleGAN2 生成器以保留与面部身份相关的几何和色彩特征。通过定性和定量评估以及消融实验，证明了我们的方法相较于现有方法的优势。