Existing 3D-aware portrait synthesis methods can generate impressive
high-quality images while preserving strong 3D consistency. However, most of
them cannot support the fine-grained part-level control over synthesized
images. Conversely, some GAN-based 2D portrait synthesis methods can achieve
clear disentanglement of facial regions, but they cannot preserve view
consistency due to a lack of 3D modeling abilities. To address these issues, we
propose 3D-SSGAN, a novel framework for 3D-aware compositional portrait image
synthesis. First, a simple yet effective depth-guided 2D-to-3D lifting module
maps the generated 2D part features and semantics to 3D. Then, a volume
renderer with a novel 3D-aware semantic mask renderer is utilized to produce
the composed face features and corresponding masks. The whole framework is
trained end-to-end by discriminating between real and synthesized 2D images and
their semantic masks. Quantitative and qualitative evaluations demonstrate the
superiority of 3D-SSGAN in controllable part-level synthesis while preserving
3D view consistency.

3D-SSGAN 是一个新颖的 3D 感知合成肖像图像的框架，通过提供细粒度的部件级控制，并保持 3D 视图的一致性，在合成肖像图像的同时实现了清晰的人脸区域解缠，其通过引入一个基于深度引导的 2D 到 3D 映射模块和一个具有新颖的 3D 感知语义掩码渲染器的体积渲染器来生成合成的人脸特征和相应的掩码，并通过区分真实和合成的 2D 图像和语义掩码的方式进行端到端的训练，定量和定性评估证明了 3D-SSGAN 在可控的部件级合成和保持 3D 视图一致性方面的优越性。

3D-SSGAN: 二维语义向三维感知组合肖像合成

3D-SSGAN: Lifting 2D Semantics for 3D-Aware Compositional Portrait  Synthesis

Recently 3D-aware GAN methods with neural radiance field have developed
rapidly. However, current methods model the whole image as an overall neural
radiance field, which limits the partial semantic editability of synthetic
results. Since NeRF renders an image pixel by pixel, it is possible to split
NeRF in the spatial dimension. We propose a Compositional Neural Radiance Field
(CNeRF) for semantic 3D-aware portrait synthesis and manipulation. CNeRF
divides the image by semantic regions and learns an independent neural radiance
field for each region, and finally fuses them and renders the complete image.
Thus we can manipulate the synthesized semantic regions independently, while
fixing the other parts unchanged. Furthermore, CNeRF is also designed to
decouple shape and texture within each semantic region. Compared to
state-of-the-art 3D-aware GAN methods, our approach enables fine-grained
semantic region manipulation, while maintaining high-quality 3D-consistent
synthesis. The ablation studies show the effectiveness of the structure and
loss function used by our method. In addition real image inversion and cartoon
portrait 3D editing experiments demonstrate the application potential of our
method.

通过分割语义区域并利用神经辐射场学习各自的辐射场，最后合并成完整图像，CNeRF 方法可以实现细粒度的语义区域操作，同时维持高质量的 3D 一致合成。