Existing methods for 3D-aware image synthesis largely depend on the 3D pose
distribution pre-estimated on the training set. An inaccurate estimation may
mislead the model into learning faulty geometry. This work proposes PoF3D that
frees generative radiance fields from the requirements of 3D pose priors. We
first equip the generator with an efficient pose learner, which is able to
infer a pose from a latent code, to approximate the underlying true pose
distribution automatically. We then assign the discriminator a task to learn
pose distribution under the supervision of the generator and to differentiate
real and synthesized images with the predicted pose as the condition. The
pose-free generator and the pose-aware discriminator are jointly trained in an
adversarial manner. Extensive results on a couple of datasets confirm that the
performance of our approach, regarding both image quality and geometry quality,
is on par with state of the art. To our best knowledge, PoF3D demonstrates the
feasibility of learning high-quality 3D-aware image synthesis without using 3D
pose priors for the first time.

本文提出了一种名为 PoF3D 的方法，通过自动学习真实姿态分布，使生成辐射场摆脱了先前在训练集上预估的 3D 姿态分布的先验要求，同时使用 pose-aware 辨别器，并通过对抗训练的方式进行学习，从而实现了不使用 3D 姿态先验条件下学习高质量的 3D-aware 图像合成。

利用未知姿态分布学习三维感知图像合成

Learning 3D-aware Image Synthesis with Unknown Pose Distribution

The advent of generative radiance fields has significantly promoted the
development of 3D-aware image synthesis. The cumulative rendering process in
radiance fields makes training these generative models much easier since
gradients are distributed over the entire volume, but leads to diffused object
surfaces. In the meantime, compared to radiance fields occupancy
representations could inherently ensure deterministic surfaces. However, if we
directly apply occupancy representations to generative models, during training
they will only receive sparse gradients located on object surfaces and
eventually suffer from the convergence problem. In this paper, we propose
Generative Occupancy Fields (GOF), a novel model based on generative radiance
fields that can learn compact object surfaces without impeding its training
convergence. The key insight of GOF is a dedicated transition from the
cumulative rendering in radiance fields to rendering with only the surface
points as the learned surface gets more and more accurate. In this way, GOF
combines the merits of two representations in a unified framework. In practice,
the training-time transition of start from radiance fields and march to
occupancy representations is achieved in GOF by gradually shrinking the
sampling region in its rendering process from the entire volume to a minimal
neighboring region around the surface. Through comprehensive experiments on
multiple datasets, we demonstrate that GOF can synthesize high-quality images
with 3D consistency and simultaneously learn compact and smooth object
surfaces. Code, models, and demo videos are available at
this https URL

本文提出了一种名为 GOF 的新模型，它基于生成亮度场，可以在不妨碍其训练收敛的情况下学习紧凑的物体表面。GOF 的关键在于从亮度场的累积渲染到只以表面点为基础的渲染的特定转换。通过对多个数据集的全面实验，我们证明了 GOF 可以生成具有 3D 一致性的高质量图像，并同时学习到紧凑而光滑的物体表面。

基于生成式占位场的三维表面感知图像合成

Generative Occupancy Fields for 3D Surface-Aware Image Synthesis

The advancement of generative radiance fields has pushed the boundary of
3D-aware image synthesis. Motivated by the observation that a 3D object should
look realistic from multiple viewpoints, these methods introduce a multi-view
constraint as regularization to learn valid 3D radiance fields from 2D images.
Despite the progress, they often fall short of capturing accurate 3D shapes due
to the shape-color ambiguity, limiting their applicability in downstream tasks.
In this work, we address this ambiguity by proposing a novel shading-guided
generative implicit model that is able to learn a starkly improved shape
representation. Our key insight is that an accurate 3D shape should also yield
a realistic rendering under different lighting conditions. This multi-lighting
constraint is realized by modeling illumination explicitly and performing
shading with various lighting conditions. Gradients are derived by feeding the
synthesized images to a discriminator. To compensate for the additional
computational burden of calculating surface normals, we further devise an
efficient volume rendering strategy via surface tracking, reducing the training
and inference time by 24% and 48%, respectively. Our experiments on multiple
datasets show that the proposed approach achieves photorealistic 3D-aware image
synthesis while capturing accurate underlying 3D shapes. We demonstrate
improved performance of our approach on 3D shape reconstruction against
existing methods, and show its applicability on image relighting. Our code will
be released at this https URL

本研究提出了一种基于阴影引导的生成式隐式模型，该模型利用多角度光照约束对 3D 物体进行建模，使得在不同光照条件下产生逼真的渲染效果，从而改善先前方法在表达精确 3D 形状方面的局限性，并成功应用于图像重照和 3D 形状重建任务。