3D occupancy prediction is an emerging task that aims to estimate the
occupancy states and semantics of 3D scenes using multi-view images. However,
image-based scene perception encounters significant challenges in achieving
accurate prediction due to the absence of geometric priors. In this paper, we
address this issue by exploring cross-modal knowledge distillation in this
task, i.e., we leverage a stronger multi-modal model to guide the visual model
during training. In practice, we observe that directly applying features or
logits alignment, proposed and widely used in bird's-eyeview (BEV) perception,
does not yield satisfactory results. To overcome this problem, we introduce
RadOcc, a Rendering assisted distillation paradigm for 3D Occupancy prediction.
By employing differentiable volume rendering, we generate depth and semantic
maps in perspective views and propose two novel consistency criteria between
the rendered outputs of teacher and student models. Specifically, the depth
consistency loss aligns the termination distributions of the rendered rays,
while the semantic consistency loss mimics the intra-segment similarity guided
by vision foundation models (VLMs). Experimental results on the nuScenes
dataset demonstrate the effectiveness of our proposed method in improving
various 3D occupancy prediction approaches, e.g., our proposed methodology
enhances our baseline by 2.2% in the metric of mIoU and achieves 50% in Occ3D
benchmark.

探索 3D 场景的占据状态和语义的三维占据预测任务中，通过利用更强大的多模态模型在训练时指导视觉模型，提出了渲染辅助蒸馏的范例 RadOcc，其中利用可微的体积渲染在透视视图中生成深度和语义图，并提出了两个新的一致性标准，即深度一致性损失和语义一致性损失，实验证明该方法在提高各种 3D 占据预测方法方面的有效性。