While recent 3D-aware generative models have shown photo-realistic image
synthesis with multi-view consistency, the synthesized image quality degrades
depending on the camera pose (e.g., a face with a blurry and noisy boundary at
a side viewpoint). Such degradation is mainly caused by the difficulty of
learning both pose consistency and photo-realism simultaneously from a dataset
with heavily imbalanced poses. In this paper, we propose SideGAN, a novel 3D
GAN training method to generate photo-realistic images irrespective of the
camera pose, especially for faces of side-view angles. To ease the challenging
problem of learning photo-realistic and pose-consistent image synthesis, we
split the problem into two subproblems, each of which can be solved more
easily. Specifically, we formulate the problem as a combination of two simple
discrimination problems, one of which learns to discriminate whether a
synthesized image looks real or not, and the other learns to discriminate
whether a synthesized image agrees with the camera pose. Based on this, we
propose a dual-branched discriminator with two discrimination branches. We also
propose a pose-matching loss to learn the pose consistency of 3D GANs. In
addition, we present a pose sampling strategy to increase learning
opportunities for steep angles in a pose-imbalanced dataset. With extensive
validation, we demonstrate that our approach enables 3D GANs to generate
high-quality geometries and photo-realistic images irrespective of the camera
pose.

通过将问题拆分为两个子问题，我们提出了 SideGAN，一种新颖的 3D GAN 训练方法，用于生成与相机位置无关的逼真图像，特别适用于侧视角的人脸。

SideGAN: 用于改进侧视图图像生成的三维感知生成模型

SideGAN: 3D-Aware Generative Model for Improved Side-View Image  Synthesis

State-of-the-art 3D-aware generative models rely on coordinate-based MLPs to
parameterize 3D radiance fields. While demonstrating impressive results,
querying an MLP for every sample along each ray leads to slow rendering.
Therefore, existing approaches often render low-resolution feature maps and
process them with an upsampling network to obtain the final image. Albeit
efficient, neural rendering often entangles viewpoint and content such that
changing the camera pose results in unwanted changes of geometry or appearance.
Motivated by recent results in voxel-based novel view synthesis, we investigate
the utility of sparse voxel grid representations for fast and 3D-consistent
generative modeling in this paper. Our results demonstrate that monolithic MLPs
can indeed be replaced by 3D convolutions when combining sparse voxel grids
with progressive growing, free space pruning and appropriate regularization. To
obtain a compact representation of the scene and allow for scaling to higher
voxel resolutions, our model disentangles the foreground object (modeled in 3D)
from the background (modeled in 2D). In contrast to existing approaches, our
method requires only a single forward pass to generate a full 3D scene. It
hence allows for efficient rendering from arbitrary viewpoints while yielding
3D consistent results with high visual fidelity.

利用稀疏像素网格表示快速且三维一致地进行生成建模，通过融合渐进式生长、自由空间修剪和适当的正则化方法将单块的 MLPs 替换成三维卷积，实现对具有高视觉保真度的三维场景的高效渲染。

VoxGRAF: 基于稀疏体素网格的快速 3D 感知图像合成

VoxGRAF: Fast 3D-Aware Image Synthesis with Sparse Voxel Grids

3D-aware generative models have shown that the introduction of 3D information
can lead to more controllable image generation. In particular, the current
state-of-the-art model GIRAFFE can control each object's rotation, translation,
scale, and scene camera pose without corresponding supervision. However,
GIRAFFE only operates well when the image resolution is low. We propose GIRAFFE
HD, a high-resolution 3D-aware generative model that inherits all of GIRAFFE's
controllable features while generating high-quality, high-resolution images
($512^2$ resolution and above). The key idea is to leverage a style-based
neural renderer, and to independently generate the foreground and background to
force their disentanglement while imposing consistency constraints to stitch
them together to composite a coherent final image. We demonstrate
state-of-the-art 3D controllable high-resolution image generation on multiple
natural image datasets.

本文提出了一种高分辨率的 3D 生成模型 ——GIRAFFE HD，该模型继承了 GIRAFFE 模型的所有可控制特性，能够生成高质量、高分辨率的图像，并且通过独立生成前景和背景，并将它们结合起来以组成一个相关性的最终图像，实现了自然图像数据集上的 3D 可控制的高分辨率图像生成。

GIRAFFE HD: 高分辨率的三维感知生成模型

GIRAFFE HD: A High-Resolution 3D-aware Generative Model

Tremendous progress in deep generative models has led to photorealistic image
synthesis. While achieving compelling results, most approaches operate in the
two-dimensional image domain, ignoring the three-dimensional nature of our
world. Several recent works therefore propose generative models which are
3D-aware, i.e., scenes are modeled in 3D and then rendered differentiably to
the image plane. This leads to impressive 3D consistency, but incorporating
such a bias comes at a price: the camera needs to be modeled as well. Current
approaches assume fixed intrinsics and a predefined prior over camera pose
ranges. As a result, parameter tuning is typically required for real-world
data, and results degrade if the data distribution is not matched. Our key
hypothesis is that learning a camera generator jointly with the image generator
leads to a more principled approach to 3D-aware image synthesis. Further, we
propose to decompose the scene into a background and foreground model, leading
to more efficient and disentangled scene representations. While training from
raw, unposed image collections, we learn a 3D- and camera-aware generative
model which faithfully recovers not only the image but also the camera data
distribution. At test time, our model generates images with explicit control
over the camera as well as the shape and appearance of the scene.

本文提出了一种深度生成模型，它是基于 3D 且与摄像机相关的，通过将场景分成背景和前景模型，并学习摄像机与图像生成器，达到了更高效且可分离的场景再现效果。