While recent advances in 3D-aware Generative Adversarial Networks (GANs) have
aided the development of near-frontal view human face synthesis, the challenge
of comprehensively synthesizing a full 3D head viewable from all angles still
persists. Although PanoHead proves the possibilities of using a large-scale
dataset with images of both frontal and back views for full-head synthesis, it
often causes artifacts for back views. Based on our in-depth analysis, we found
the reasons are mainly twofold. First, from network architecture perspective,
we found each plane in the utilized tri-plane/tri-grid representation space
tends to confuse the features from both sides, causing "mirroring" artifacts
(e.g., the glasses appear in the back). Second, from data supervision aspect,
we found that existing discriminator training in 3D GANs mainly focuses on the
quality of the rendered image itself, and does not care much about its
plausibility with the perspective from which it was rendered. This makes it
possible to generate "face" in non-frontal views, due to its easiness to fool
the discriminator. In response, we propose SphereHead, a novel tri-plane
representation in the spherical coordinate system that fits the human head's
geometric characteristics and efficiently mitigates many of the generated
artifacts. We further introduce a view-image consistency loss for the
discriminator to emphasize the correspondence of the camera parameters and the
images. The combination of these efforts results in visually superior outcomes
with significantly fewer artifacts. Our code and dataset are publicly available
at this https URL

通过使用球面坐标系中的新型三平面表示，以及引入视图 - 图像一致性损失来减少生成的伪影，该研究提出了一种解决全方位，全角度三维头部合成挑战的方法。

SphereHead: 基于球面三平面表示的稳定三维全头合成

SphereHead: Stable 3D Full-head Synthesis with Spherical Tri-plane  Representation

Creating controllable, photorealistic, and geometrically detailed digital
doubles of real humans solely from video data is a key challenge in Computer
Graphics and Vision, especially when real-time performance is required. Recent
methods attach a neural radiance field (NeRF) to an articulated structure,
e.g., a body model or a skeleton, to map points into a pose canonical space
while conditioning the NeRF on the skeletal pose. These approaches typically
parameterize the neural field with a multi-layer perceptron (MLP) leading to a
slow runtime. To address this drawback, we propose TriHuman a novel
human-tailored, deformable, and efficient tri-plane representation, which
achieves real-time performance, state-of-the-art pose-controllable geometry
synthesis as well as photorealistic rendering quality. At the core, we
non-rigidly warp global ray samples into our undeformed tri-plane texture
space, which effectively addresses the problem of global points being mapped to
the same tri-plane locations. We then show how such a tri-plane feature
representation can be conditioned on the skeletal motion to account for dynamic
appearance and geometry changes. Our results demonstrate a clear step towards
higher quality in terms of geometry and appearance modeling of humans as well
as runtime performance.

通过视频数据创建可控、逼真、几何详细的数字化人类复制品是计算机图形学和视觉领域的关键挑战，特别是当需要实时性能时。最近的方法将神经光辐射场（NeRF）附加到关节结构上，如身体模型或骨骼，将点映射到姿势规范空间，同时条件化 NeRF 的骨骼姿势。我们提出了 TriHuman，一个新的人体定制、可变形和高效的三平面表示方法，实现了实时性能、最先进的姿势控制几何合成以及逼真的渲染质量。在核心部分中，我们非刚性地将全局光线采样变形到无变形的三平面纹理空间中，这有效解决了将全局点映射到相同三平面位置的问题。然后，我们展示了如何将这种三平面特征表示方法与骨骼运动相关联，以考虑动态外观和几何变化。我们的结果表明，对于人体的几何和外观建模以及运行时间性能，有了明显的质量提升。

TriHuman: 实时可控的三平面表示法用于详细人体几何和外观合成

TriHuman : A Real-time and Controllable Tri-plane Representation for  Detailed Human Geometry and Appearance Synthesis

A signed distance function (SDF) parametrized by an MLP is a common
ingredient of neural surface reconstruction. We build on the successful recent
method NeuS to extend it by three new components. The first component is to
borrow the tri-plane representation from EG3D and represent signed distance
fields as a mixture of tri-planes and MLPs instead of representing it with MLPs
only. Using tri-planes leads to a more expressive data structure but will also
introduce noise in the reconstructed surface. The second component is to use a
new type of positional encoding with learnable weights to combat noise in the
reconstruction process. We divide the features in the tri-plane into multiple
frequency scales and modulate them with sin and cos functions of different
frequencies. The third component is to use learnable convolution operations on
the tri-plane features using self-attention convolution to produce features
with different frequency bands. The experiments show that PET-NeuS achieves
high-fidelity surface reconstruction on standard datasets. Following previous
work and using the Chamfer metric as the most important way to measure surface
reconstruction quality, we are able to improve upon the NeuS baseline by 57% on
Nerf-synthetic (0.84 compared to 1.97) and by 15.5% on DTU (0.71 compared to
0.84). The qualitative evaluation reveals how our method can better control the
interference of high-frequency noise. Code available at
https://github.com/yiqun-wang/PET-NeuS.

本文建立新的神经表面重建方法，在 signed distance function 的基础上结合 tri-plane 表示，采用可学习的位置编码和卷积操作，通过实验在标准数据集上大幅提高了表面重建的精度。

PET-NeuS: 神经表面的位置编码三平面

PET-NeuS: Positional Encoding Tri-Planes for Neural Surfaces

Recent progress in NeRF-based GANs has introduced a number of approaches for
high-resolution and high-fidelity generative modeling of human heads with a
possibility for novel view rendering. At the same time, one must solve an
inverse problem to be able to re-render or modify an existing image or video.
Despite the success of universal optimization-based methods for 2D GAN
inversion, those, applied to 3D GANs, may fail to produce 3D-consistent
renderings. Fast encoder-based techniques, such as those developed for
StyleGAN, may also be less appealing due to the lack of identity preservation.
In our work, we introduce a real-time method that bridges the gap between the
two approaches by directly utilizing the tri-plane representation introduced
for EG3D generative model. In particular, we build upon a feed-forward
convolutional encoder for the latent code and extend it with a
fully-convolutional predictor of tri-plane numerical offsets. As shown in our
work, the renderings are similar in quality to optimization-based techniques
and significantly outperform the baselines for novel view. As we empirically
prove, this is a consequence of directly operating in the tri-plane space, not
in the GAN parameter space, while making use of an encoder-based trainable
approach.

该论文介绍了一种基于三平面表示的实时方法，该方法直接利用编码器进行潜在编码，并将其扩展为三平面数值偏移的全卷积预测器，从而弥合了当今用于高质量变形和重新渲染存在的方法之间的差距。