Most existing human rendering methods require every part of the human to be
fully visible throughout the input video. However, this assumption does not
hold in real-life settings where obstructions are common, resulting in only
partial visibility of the human. Considering this, we present OccFusion, an
approach that utilizes efficient 3D Gaussian splatting supervised by pretrained
2D diffusion models for efficient and high-fidelity human rendering. We propose
a pipeline consisting of three stages. In the Initialization stage, complete
human masks are generated from partial visibility masks. In the Optimization
stage, 3D human Gaussians are optimized with additional supervision by
Score-Distillation Sampling (SDS) to create a complete geometry of the human.
Finally, in the Refinement stage, in-context inpainting is designed to further
improve rendering quality on the less observed human body parts. We evaluate
OccFusion on ZJU-MoCap and challenging OcMotion sequences and find that it
achieves state-of-the-art performance in the rendering of occluded humans.

利用高效的 3D 高斯喷粒方法，由预训练的 2D 扩散模型进行监督，提出了 OccFusion，一种用于高效和高保真度人体渲染的方法。通过三个阶段的流程，即初始化阶段、优化阶段和改进阶段，实现了对人体的完整渲染，并在实验中表现出了在渲染遮挡人物方面具有领先水平的性能。

OccFusion：利用生成扩散先验渲染被遮挡的人体

OccFusion: Rendering Occluded Humans with Generative Diffusion Priors

3D occupancy prediction based on multi-sensor fusion, crucial for a reliable
autonomous driving system, enables fine-grained understanding of 3D scenes.
Previous fusion-based 3D occupancy predictions relied on depth estimation for
processing 2D image features. However, depth estimation is an ill-posed
problem, hindering the accuracy and robustness of these methods. Furthermore,
fine-grained occupancy prediction demands extensive computational resources. We
introduce OccFusion, a multi-modal fusion method free from depth estimation,
and a corresponding point cloud sampling algorithm for dense integration of
image features. Building on this, we propose an active training method and an
active coarse to fine pipeline, enabling the model to adaptively learn more
from complex samples and optimize predictions specifically for challenging
areas such as small or overlapping objects. The active methods we propose can
be naturally extended to any occupancy prediction model. Experiments on the
OpenOccupancy benchmark show our method surpasses existing state-of-the-art
(SOTA) multi-modal methods in IoU across all categories. Additionally, our
model is more efficient during both the training and inference phases,
requiring far fewer computational resources. Comprehensive ablation studies
demonstrate the effectiveness of our proposed techniques.

基于多传感器融合的 3D 占用预测方法 OccFusion，使用无需深度估计的多模态融合以及相应的点云采样算法，通过主动训练方法和主动粗到细流程，提高了复杂场景中预测的准确性和鲁棒性，有效优化了计算资源需求。在开放占用评测中，该方法超过了现有基于多模态的最先进方法，并且训练和推理阶段更加高效。细致的消融研究证明了我们提出的技术的有效性。

OccFusion: 无深度估计的多传感器融合用于 3D 占据预测

OccFusion: Depth Estimation Free Multi-sensor Fusion for 3D Occupancy  Prediction

This paper introduces OccFusion, a straightforward and efficient sensor
fusion framework for predicting 3D occupancy. A comprehensive understanding of
3D scenes is crucial in autonomous driving, and recent models for 3D semantic
occupancy prediction have successfully addressed the challenge of describing
real-world objects with varied shapes and classes. However, existing methods
for 3D occupancy prediction heavily rely on surround-view camera images, making
them susceptible to changes in lighting and weather conditions. By integrating
features from additional sensors, such as lidar and surround view radars, our
framework enhances the accuracy and robustness of occupancy prediction,
resulting in top-tier performance on the nuScenes benchmark. Furthermore,
extensive experiments conducted on the nuScenes dataset, including challenging
night and rainy scenarios, confirm the superior performance of our sensor
fusion strategy across various perception ranges. The code for this framework
will be made available at this https URL

该研究论文介绍了 OccFusion，一种直观高效的传感器融合框架，可用于预测三维占据情况。通过整合来自其他传感器（如激光雷达和环绕视图雷达）的特征，该框架提高了占据情况预测的准确性和稳健性，在 nuScenes 基准测试中取得了顶尖性能。