Rendering dynamic 3D human from monocular videos is crucial for various
applications such as virtual reality and digital entertainment. Most methods
assume the people is in an unobstructed scene, while various objects may cause
the occlusion of body parts in real-life scenarios. Previous method utilizing
NeRF for surface rendering to recover the occluded areas, but it requiring more
than one day to train and several seconds to render, failing to meet the
requirements of real-time interactive applications. To address these issues, we
propose OccGaussian based on 3D Gaussian Splatting, which can be trained within
6 minutes and produces high-quality human renderings up to 160 FPS with
occluded input. OccGaussian initializes 3D Gaussian distributions in the
canonical space, and we perform occlusion feature query at occluded regions,
the aggregated pixel-align feature is extracted to compensate for the missing
information. Then we use Gaussian Feature MLP to further process the feature
along with the occlusion-aware loss functions to better perceive the occluded
area. Extensive experiments both in simulated and real-world occlusions,
demonstrate that our method achieves comparable or even superior performance
compared to the state-of-the-art method. And we improving training and
inference speeds by 250x and 800x, respectively. Our code will be available for
research purposes.

我们提出了基于 3D 高斯散射的 OccGaussian 方法，其可在 6 分钟内进行训练，并且以高质量 160FPS 渲染具有遮挡输入的人体场景。通过在标准空间中初始化 3D 高斯分布，并在遮挡区域进行遮挡特征查询，提取聚合的像素对齐特征来弥补丢失的信息，然后我们使用高斯特征 MLP 进一步处理特征，并结合面向遮挡区域的损失函数来更好地感知遮挡区域。大量的实验结果表明，我们的方法在模拟和真实世界遮挡场景中实现了与最先进方法相媲美甚至更优的性能，且训练和推理速度提高了 250 倍和 800 倍，我们的代码将提供给研究目的使用。

OccGaussian：用于遮挡人体渲染的三维高斯粒子

OccGaussian: 3D Gaussian Splatting for Occluded Human Rendering

In this work, we propose a method to address the challenge of rendering a 3D
human from a single image in a free-view manner. Some existing approaches could
achieve this by using generalizable pixel-aligned implicit fields to
reconstruct a textured mesh of a human or by employing a 2D diffusion model as
guidance with the Score Distillation Sampling (SDS) method, to lift the 2D
image into 3D space. However, a generalizable implicit field often results in
an over-smooth texture field, while the SDS method tends to lead to a
texture-inconsistent novel view with the input image. In this paper, we
introduce a texture-consistent back view synthesis module that could transfer
the reference image content to the back view through depth and text-guided
attention injection. Moreover, to alleviate the color distortion that occurs in
the side region, we propose a visibility-aware patch consistency regularization
for texture mapping and refinement combined with the synthesized back view
texture. With the above techniques, we could achieve high-fidelity and
texture-consistent human rendering from a single image. Experiments conducted
on both real and synthetic data demonstrate the effectiveness of our method and
show that our approach outperforms previous baseline methods.

本文提出了一种从单幅图像中渲染出具有一致纹理的 3D 人物的方法，并通过深度和文本引导的关注注入将参考图像内容转移到背景视图，同时结合合成的背景视图纹理实现可见性感知的贴图和细化，通过实验证明了方法的有效性并展示了超越基准方法的成果。

ConTex-Human: 从单幅图像中进行纹理一致合成的人类自由视点渲染

ConTex-Human: Free-View Rendering of Human from a Single Image with  Texture-Consistent Synthesis

There has been rapid progress recently on 3D human rendering, including novel
view synthesis and pose animation, based on the advances of neural radiance
fields (NeRF). However, most existing methods focus on person-specific training
and their training typically requires multi-view videos. This paper deals with
a new challenging task -- rendering novel views and novel poses for a person
unseen in training, using only multiview images as input. For this task, we
propose a simple yet effective method to train a generalizable NeRF with
multiview images as conditional input. The key ingredient is a dedicated
representation combining a canonical NeRF and a volume deformation scheme.
Using a canonical space enables our method to learn shared properties of human
and easily generalize to different people. Volume deformation is used to
connect the canonical space with input and target images and query image
features for radiance and density prediction. We leverage the parametric 3D
human model fitted on the input images to derive the deformation, which works
quite well in practice when combined with our canonical NeRF. The experiments
on both real and synthetic data with the novel view synthesis and pose
animation tasks collectively demonstrate the efficacy of our method.

本文提出了一种简单而有效的方法，使用多视角图像作为条件输入，训练出适用于不同人的可泛化 NeRF，该方法的关键部分是将规范空间与输入和目标图像相连，并利用参数化的 3D 人体模型来导出变形，本方法在实现新视角合成和姿势动画任务方面表现出了很好的效果。