As a fundamental task of vision-based perception, 3D occupancy prediction
reconstructs 3D structures of surrounding environments. It provides detailed
information for autonomous driving planning and navigation. However, most
existing methods heavily rely on the LiDAR point clouds to generate occupancy
ground truth, which is not available in the vision-based system. In this paper,
we propose an OccNeRF method for self-supervised multi-camera occupancy
prediction. Different from bounded 3D occupancy labels, we need to consider
unbounded scenes with raw image supervision. To solve the issue, we
parameterize the reconstructed occupancy fields and reorganize the sampling
strategy. The neural rendering is adopted to convert occupancy fields to
multi-camera depth maps, supervised by multi-frame photometric consistency.
Moreover, for semantic occupancy prediction, we design several strategies to
polish the prompts and filter the outputs of a pretrained open-vocabulary 2D
segmentation model. Extensive experiments for both self-supervised depth
estimation and semantic occupancy prediction tasks on nuScenes dataset
demonstrate the effectiveness of our method.

我们提出了一种自我监督的多摄像机占用预测 OccNeRF 方法，用于三维占用预测任务和语义占用预测任务。

OccNeRF: 基于神经辐射场的自监督多相机占用预测

OccNeRF: Self-Supervised Multi-Camera Occupancy Prediction with Neural  Radiance Fields

Semantic occupancy prediction aims to infer dense geometry and semantics of
surroundings for an autonomous agent to operate safely in the 3D environment.
Existing occupancy prediction methods are almost entirely trained on
human-annotated volumetric data. Although of high quality, the generation of
such 3D annotations is laborious and costly, restricting them to a few specific
object categories in the training dataset. To address this limitation, this
paper proposes Open Vocabulary Occupancy (OVO), a novel approach that allows
semantic occupancy prediction of arbitrary classes but without the need for 3D
annotations during training. Keys to our approach are (1) knowledge
distillation from a pre-trained 2D open-vocabulary segmentation model to the 3D
occupancy network, and (2) pixel-voxel filtering for high-quality training data
generation. The resulting framework is simple, compact, and compatible with
most state-of-the-art semantic occupancy prediction models. On NYUv2 and
SemanticKITTI datasets, OVO achieves competitive performance compared to
supervised semantic occupancy prediction approaches. Furthermore, we conduct
extensive analyses and ablation studies to offer insights into the design of
the proposed framework.

本文提出了 Open Vocabulary Occupancy (OVO) 算法，通过知识蒸馏和像素 - 体素筛选两个关键步骤，可以对任意类别进行语义占据的预测，同时适用于大多数最先进的语义占据预测模型。在 NYUv2 和 SemanticKITTI 数据集上，OVO 与有监督的方法相比具有竞争性的性能。

OVO: 开放词汇占用

OVO: Open-Vocabulary Occupancy

The vision-based perception for autonomous driving has undergone a
transformation from the bird-eye-view (BEV) representations to the 3D semantic
occupancy. Compared with the BEV planes, the 3D semantic occupancy further
provides structural information along the vertical direction. This paper
presents OccFormer, a dual-path transformer network to effectively process the
3D volume for semantic occupancy prediction. OccFormer achieves a long-range,
dynamic, and efficient encoding of the camera-generated 3D voxel features. It
is obtained by decomposing the heavy 3D processing into the local and global
transformer pathways along the horizontal plane. For the occupancy decoder, we
adapt the vanilla Mask2Former for 3D semantic occupancy by proposing
preserve-pooling and class-guided sampling, which notably mitigate the sparsity
and class imbalance. Experimental results demonstrate that OccFormer
significantly outperforms existing methods for semantic scene completion on
SemanticKITTI dataset and for LiDAR semantic segmentation on nuScenes dataset.
Code is available at https://github.com/zhangyp15/OccFormer.

本文介绍了 OccFormer，一种双路径 Transformer 网络，有效地处理了用于自主驾驶的 3D 立体体素特征，并在 SemanticKITTI 数据集上对语义完整性和 nuScenes 数据集上的 LiDAR 语义分割方面优于现有方法。