Most existing human rendering methods require every part of the human to be
fully visible throughout the input video. However, this assumption does not
hold in real-life settings where obstructions are common, resulting in only
partial visibility of the human. Considering this, we present OccFusion, an
approach that utilizes efficient 3D Gaussian splatting supervised by pretrained
2D diffusion models for efficient and high-fidelity human rendering. We propose
a pipeline consisting of three stages. In the Initialization stage, complete
human masks are generated from partial visibility masks. In the Optimization
stage, 3D human Gaussians are optimized with additional supervision by
Score-Distillation Sampling (SDS) to create a complete geometry of the human.
Finally, in the Refinement stage, in-context inpainting is designed to further
improve rendering quality on the less observed human body parts. We evaluate
OccFusion on ZJU-MoCap and challenging OcMotion sequences and find that it
achieves state-of-the-art performance in the rendering of occluded humans.

利用高效的 3D 高斯喷粒方法，由预训练的 2D 扩散模型进行监督，提出了 OccFusion，一种用于高效和高保真度人体渲染的方法。通过三个阶段的流程，即初始化阶段、优化阶段和改进阶段，实现了对人体的完整渲染，并在实验中表现出了在渲染遮挡人物方面具有领先水平的性能。

OccFusion：利用生成扩散先验渲染被遮挡的人体

OccFusion: Rendering Occluded Humans with Generative Diffusion Priors

This paper addresses the problem of human rendering in the video with
temporal appearance constancy. Reconstructing dynamic body shapes with
volumetric neural rendering methods, such as NeRF, requires finding the
correspondence of the points in the canonical and observation space, which
demands understanding human body shape and motion. Some methods use rigid
transformation, such as SE(3), which cannot precisely model each frame's unique
motion and muscle movements. Others generate the transformation for each frame
with a trainable network, such as neural blend weight field or translation
vector field, which does not consider the appearance constancy of general body
shape. In this paper, we propose CAT-NeRF for self-awareness of appearance
constancy with Tx$^2$Former, a novel way to combine two Transformer layers, to
separate appearance constancy and uniqueness. Appearance constancy models the
general shape across the video, and uniqueness models the unique patterns for
each frame. We further introduce a novel Covariance Loss to limit the
correlation between each pair of appearance uniquenesses to ensure the
frame-unique pattern is maximally captured in appearance uniqueness. We assess
our method on H36M and ZJU-MoCap and show state-of-the-art performance.

本文提出了一种称为 CAT-NeRF 的新方法，用于人体渲染视频中的外观一致性自我感知，并使用 Tx^2Former 来将外观一致性和独特性分离。同时，引入了新的 Covariance Loss 来限制外观独特性之间的相关性，以确保外观独特性能够被最大程度地捕捉。以 H36M 和 ZJU-MoCap 为例，实验结果显示这种方法具有最先进的性能。