Robotic perception requires the modeling of both 3D geometry and semantics.
Existing methods typically focus on estimating 3D bounding boxes, neglecting
finer geometric details and struggling to handle general, out-of-vocabulary
objects. To overcome these limitations, we introduce a novel task for 3D
occupancy prediction, which aims to estimate the detailed occupancy and
semantics of objects from multi-view images. To facilitate this task, we
develop a label generation pipeline that produces dense, visibility-aware
labels for a given scene. This pipeline includes point cloud aggregation, point
labeling, and occlusion handling. We construct two benchmarks based on the
Waymo Open Dataset and the nuScenes Dataset, resulting in the Occ3D-Waymo and
Occ3D-nuScenes benchmarks. Lastly, we propose a model, dubbed Coarse-to-Fine
Occupancy (CTF-Occ) network, which demonstrates superior performance in the 3D
occupancy prediction task. This approach addresses the need for finer geometric
understanding in a coarse-to-fine fashion. The code, data, and benchmarks are
released at this https URL

本研究提出了一种新的 3D 占据预测任务，旨在从多视图图像中估计对象的详细占据和语义信息，并介绍了 Coarse-to-Fine Occupancy (CTF-Occ) 网络模型，该模型在 3D 占据预测任务中表现出优越的性能。