NeRFs have enabled highly realistic synthesis of human faces including
complex appearance and reflectance effects of hair and skin. These methods
typically require a large number of multi-view input images, making the process
hardware intensive and cumbersome, limiting applicability to unconstrained
settings. We propose a novel volumetric human face prior that enables the
synthesis of ultra high-resolution novel views of subjects that are not part of
the prior's training distribution. This prior model consists of an
identity-conditioned NeRF, trained on a dataset of low-resolution multi-view
images of diverse humans with known camera calibration. A simple sparse
landmark-based 3D alignment of the training dataset allows our model to learn a
smooth latent space of geometry and appearance despite a limited number of
training identities. A high-quality volumetric representation of a novel
subject can be obtained by model fitting to 2 or 3 camera views of arbitrary
resolution. Importantly, our method requires as few as two views of casually
captured images as input at inference time.

提出了一种新颖的体积化人脸先验模型来合成不同于训练分布的主体的超高分辨率新视角，仅需要两个随意拍摄图像的输入视图。

基于数据驱动的体积先验模型用于少样本超高分辨率人脸合成

Preface: A Data-driven Volumetric Prior for Few-shot Ultra  High-resolution Face Synthesis

Recent volumetric 3D reconstruction methods can produce very accurate
results, with plausible geometry even for unobserved surfaces. However, they
face an undesirable trade-off when it comes to multi-view fusion. They can fuse
all available view information by global averaging, thus losing fine detail, or
they can heuristically cluster views for local fusion, thus restricting their
ability to consider all views jointly. Our key insight is that greater detail
can be retained without restricting view diversity by learning a view-fusion
function conditioned on camera pose and image content. We propose to learn this
multi-view fusion using a transformer. To this end, we introduce VoRTX, an
end-to-end volumetric 3D reconstruction network using transformers for
wide-baseline, multi-view feature fusion. Our model is occlusion-aware,
leveraging the transformer architecture to predict an initial, projective scene
geometry estimate. This estimate is used to avoid backprojecting image features
through surfaces into occluded regions. We train our model on ScanNet and show
that it produces better reconstructions than state-of-the-art methods. We also
demonstrate generalization without any fine-tuning, outperforming the same
state-of-the-art methods on two other datasets, TUM-RGBD and ICL-NUIM.

通过学习条件于相机姿态和图像内容的视图融合功能，利用变压器提出端到端体积三维重建网络 VoRTX，其模型鲁棒性强，处理视角多样，能保留更多的细节信息，比现有最先进方法更出色。

使用 Transformer 进行体素级别视角选择和融合的体积式 3D 重建

VoRTX: Volumetric 3D Reconstruction With Transformers for Voxelwise View  Selection and Fusion

We propose DeepHuman, an image-guided volume-to-volume translation CNN for 3D
human reconstruction from a single RGB image. To reduce the ambiguities
associated with the surface geometry reconstruction, even for the
reconstruction of invisible areas, we propose and leverage a dense semantic
representation generated from SMPL model as an additional input. One key
feature of our network is that it fuses different scales of image features into
the 3D space through volumetric feature transformation, which helps to recover
accurate surface geometry. The visible surface details are further refined
through a normal refinement network, which can be concatenated with the volume
generation network using our proposed volumetric normal projection layer. We
also contribute THuman, a 3D real-world human model dataset containing about
7000 models. The network is trained using training data generated from the
dataset. Overall, due to the specific design of our network and the diversity
in our dataset, our method enables 3D human model estimation given only a
single image and outperforms state-of-the-art approaches.

提出了一种用于从单个 RGB 图像进行 3D 人体重建的基于图像引导的体积到体积的转换 CNN，通过 SMPL 模型生成密集的语义表示来减少与表面几何重建相关的歧义，将不同尺度的图像特征融合到三维空间中，并通过法线细化网络来进一步提高可见面细节的精度，使用所提出的体积法向投影层将其连接到体积生成网络中，并使用其中包含的 3D 真实世界人体模型数据集进行训练，实验表明该方法在前沿方法中表现优异。

DeepHuman：从单张图像重建 3D 人体

DeepHuman: 3D Human Reconstruction from a Single Image

With the advent of affordable depth sensors, 3D capture becomes more and more
ubiquitous and already has made its way into commercial products. Yet,
capturing the geometry or complete shapes of everyday objects using scanning
devices (e.g. Kinect) still comes with several challenges that result in noise
or even incomplete shapes. Recent success in deep learning has shown how to
learn complex shape distributions in a data-driven way from large scale 3D CAD
Model collections and to utilize them for 3D processing on volumetric
representations and thereby circumventing problems of topology and
tessellation. Prior work has shown encouraging results on problems ranging from
shape completion to recognition. We provide an analysis of such approaches and
discover that training as well as the resulting representation are strongly and
unnecessarily tied to the notion of object labels. Thus, we propose a full
convolutional volumetric auto encoder that learns volumetric representation
from noisy data by estimating the voxel occupancy grids. The proposed method
outperforms prior work on challenging tasks like denoising and shape
completion. We also show that the obtained deep embedding gives competitive
performance when used for classification and promising results for shape
interpolation.

通过提出一种新的全卷积体积自动编码器，有效实现了从嘈杂的 3D 数据中提取体积表示，并在噪声去除和形状完成等多个任务上超越了以往的技术，同时在分类和形状插值方面也得到了不错的结果。