Comprehensive modeling of the surrounding 3D world is key to the success of
autonomous driving. However, existing perception tasks like object detection,
road structure segmentation, depth & elevation estimation, and open-set object
localization each only focus on a small facet of the holistic 3D scene
understanding task. This divide-and-conquer strategy simplifies the algorithm
development procedure at the cost of losing an end-to-end unified solution to
the problem. In this work, we address this limitation by studying camera-based
3D panoptic segmentation, aiming to achieve a unified occupancy representation
for camera-only 3D scene understanding. To achieve this, we introduce a novel
method called PanoOcc, which utilizes voxel queries to aggregate spatiotemporal
information from multi-frame and multi-view images in a coarse-to-fine scheme,
integrating feature learning and scene representation into a unified occupancy
representation. We have conducted extensive ablation studies to verify the
effectiveness and efficiency of the proposed method. Our approach achieves new
state-of-the-art results for camera-based semantic segmentation and panoptic
segmentation on the nuScenes dataset. Furthermore, our method can be easily
extended to dense occupancy prediction and has shown promising performance on
the Occ3D benchmark. The code will be released at
this https URL

本文介绍了一种名为 PanoOcc 的方法，它是一个基于相机的 3D 全景分割方法，它使用体素查询来聚合来自多帧和多视角图像的时空信息，并将特征学习和场景表示集成到一种全面的占用表示中，用于摄像机 3D 场景理解的统一占用表示，从而实现了更好的摄像机语义分割和全景分割结果，并且该方法可以很容易地扩展到密集的占用预测。