The function of constructing the hierarchy of objects is important to the
visual process of the human brain. Previous studies have successfully adopted
capsule networks to decompose the digits and faces into parts in an
unsupervised manner to investigate the similar perception mechanism of neural
networks. However, their descriptions are restricted to the 2D space, limiting
their capacities to imitate the intrinsic 3D perception ability of humans. In
this paper, we propose an Inverse Graphics Capsule Network (IGC-Net) to learn
the hierarchical 3D face representations from large-scale unlabeled images. The
core of IGC-Net is a new type of capsule, named graphics capsule, which
represents 3D primitives with interpretable parameters in computer graphics
(CG), including depth, albedo, and 3D pose. Specifically, IGC-Net first
decomposes the objects into a set of semantic-consistent part-level
descriptions and then assembles them into object-level descriptions to build
the hierarchy. The learned graphics capsules reveal how the neural networks,
oriented at visual perception, understand faces as a hierarchy of 3D models.
Besides, the discovered parts can be deployed to the unsupervised face
segmentation task to evaluate the semantic consistency of our method. Moreover,
the part-level descriptions with explicit physical meanings provide insight
into the face analysis that originally runs in a black box, such as the
importance of shape and texture for face recognition. Experiments on CelebA,
BP4D, and Multi-PIE demonstrate the characteristics of our IGC-Net.

本文提出了逆向图形胶囊网络 (IGC-Net)，通过图形胶囊这种新型胶囊进行层级建模和无监督学习，从大规模无标注图像中学习人脸的层级 3D 表示。通过该网络的学习，揭示了神经网络如何将人脸理解为 3D 模型的层次结构。此外，利用学习到的部分可以对无监督的人脸分割任务进行评估，并且具有明确物理含义的部分级描述提供了一些有关人脸分析的洞见。

图形胶囊：从 2D 图像学习层级的 3D 人脸表示

Graphics Capsule: Learning Hierarchical 3D Face Representations from 2D  Images

While previous speech-driven talking face generation methods have made
significant progress in improving the visual quality and lip-sync quality of
the synthesized videos, they pay less attention to lip motion jitters which
greatly undermine the realness of talking face videos. What causes motion
jitters, and how to mitigate the problem? In this paper, we conduct systematic
analyses on the motion jittering problem based on a state-of-the-art pipeline
that uses 3D face representations to bridge the input audio and output video,
and improve the motion stability with a series of effective designs. We find
that several issues can lead to jitters in synthesized talking face video: 1)
jitters from the input 3D face representations; 2) training-inference mismatch;
3) lack of dependency modeling among video frames. Accordingly, we propose
three effective solutions to address this issue: 1) we propose a gaussian-based
adaptive smoothing module to smooth the 3D face representations to eliminate
jitters in the input; 2) we add augmented erosions on the input data of the
neural renderer in training to simulate the distortion in inference to reduce
mismatch; 3) we develop an audio-fused transformer generator to model
dependency among video frames. Besides, considering there is no off-the-shelf
metric for measuring motion jitters in talking face video, we devise an
objective metric (Motion Stability Index, MSI), to quantitatively measure the
motion jitters by calculating the reciprocal of variance acceleration.
Extensive experimental results show the superiority of our method on
motion-stable face video generation, with better quality than previous systems.

本文通过对 3D 面部表示的优化和提出基于高斯自适应平滑模块和音频融合变换器生成器的解决方案，实现了稳定的人脸视频生成，并提出了一种目标度量标准 Motion Stability Index 进行运动抖动的定量测量。经过广泛的实验验证和比较，这种方法的质量明显优于以前的系统。

StableFace: 分析和改进说话人脸生成的动作稳定性

StableFace: Analyzing and Improving Motion Stability for Talking Face Generation

Recent attempts to solve the problem of head reenactment using a single
reference image have shown promising results. However, most of them either
perform poorly in terms of photo-realism, or fail to meet the identity
preservation problem, or do not fully transfer the driving pose and expression.
We propose HeadGAN, a novel system that conditions synthesis on 3D face
representations, which can be extracted from any driving video and adapted to
the facial geometry of any reference image, disentangling identity from
expression. We further improve mouth movements, by utilising audio features as
a complementary input. The 3D face representation enables HeadGAN to be further
used as an efficient method for compression and reconstruction and a tool for
expression and pose editing.

本文提出了一种基于 3D 面部表示的 HeadGAN 新系统，可以从任何驱动视频中提取，并适应于任何参考图像的面部几何形状，从而解决头部再现问题。通过利用音频特征作为补充输入，进一步改善嘴部运动，并将身份与表情分离，使其具有压缩和重建、表情和姿势编辑的功能。