Camera-based bird-eye-view (BEV) perception paradigm has made significant
progress in the autonomous driving field. Under such a paradigm, accurate BEV
representation construction relies on reliable depth estimation for
multi-camera images. However, existing approaches exhaustively predict depths
for every pixel without prioritizing objects, which are precisely the entities
requiring detection in the 3D space. To this end, we propose IA-BEV, which
integrates image-plane instance awareness into the depth estimation process
within a BEV-based detector. First, a category-specific structural priors
mining approach is proposed for enhancing the efficacy of monocular depth
generation. Besides, a self-boosting learning strategy is further proposed to
encourage the model to place more emphasis on challenging objects in
computation-expensive temporal stereo matching. Together they provide advanced
depth estimation results for high-quality BEV features construction, benefiting
the ultimate 3D detection. The proposed method achieves state-of-the-art
performances on the challenging nuScenes benchmark, and extensive experimental
results demonstrate the effectiveness of our designs.

提出了一种新方法 IA-BEV，它将图像平面实例感知集成到基于 BEV 的深度估计过程中，通过增强单目深度生成的有效性和鼓励模型在计算密集的时间立体匹配中更加注重具有挑战性的物体，为高质量 BEV 特征构建提供先进的深度估计结果，从而有利于最终的三维检测。所提出的方法在具有挑战性的 nuScenes 基准测试中取得了最先进的性能，广泛的实验结果表明了我们设计的有效性。

具有结构先验的多摄像头实例感知 3D 目标检测与自我增强学习

Instance-aware Multi-Camera 3D Object Detection with Structural Priors  Mining and Self-Boosting Learning

Multi-modal 3D scene understanding has gained considerable attention due to
its wide applications in many areas, such as autonomous driving and
human-computer interaction. Compared to conventional single-modal 3D
understanding, introducing an additional modality not only elevates the
richness and precision of scene interpretation but also ensures a more robust
and resilient understanding. This becomes especially crucial in varied and
challenging environments where solely relying on 3D data might be inadequate.
While there has been a surge in the development of multi-modal 3D methods over
past three years, especially those integrating multi-camera images (3D+2D) and
textual descriptions (3D+language), a comprehensive and in-depth review is
notably absent. In this article, we present a systematic survey of recent
progress to bridge this gap. We begin by briefly introducing a background that
formally defines various 3D multi-modal tasks and summarizes their inherent
challenges. After that, we present a novel taxonomy that delivers a thorough
categorization of existing methods according to modalities and tasks, exploring
their respective strengths and limitations. Furthermore, comparative results of
recent approaches on several benchmark datasets, together with insightful
analysis, are offered. Finally, we discuss the unresolved issues and provide
several potential avenues for future research.

本文对多模态 3D 场景理解的最新进展进行了系统调查，介绍了各种多模态任务的背景和困难，分类了现有方法，并对它们的优势和限制进行了探索，提供了在几个基准数据集上的对比结果和深入分析，最后讨论了未解决的问题并提出未来研究的几个潜在方向。

多模式三维场景理解的最新进展：综合调研和评估

Recent Advances in Multi-modal 3D Scene Understanding: A Comprehensive  Survey and Evaluation

3D visual perception tasks based on multi-camera images are essential for
autonomous driving systems. Latest work in this field performs 3D object
detection by leveraging multi-view images as an input and iteratively enhancing
object queries (object proposals) by cross-attending multi-view features.
However, individual backbone features are not updated with multi-view features
and it stays as a mere collection of the output of the single-image backbone
network. Therefore we propose 3M3D: A Multi-view, Multi-path,
Multi-representation for 3D Object Detection where we update both multi-view
features and query features to enhance the representation of the scene in both
fine panoramic view and coarse global view. Firstly, we update multi-view
features by multi-view axis self-attention. It will incorporate panoramic
information in the multi-view features and enhance understanding of the global
scene. Secondly, we update multi-view features by self-attention of the ROI
(Region of Interest) windows which encodes local finer details in the features.
It will help exchange the information not only along the multi-view axis but
also along the other spatial dimension. Lastly, we leverage the fact of
multi-representation of queries in different domains to further boost the
performance. Here we use sparse floating queries along with dense BEV (Bird's
Eye View) queries, which are later post-processed to filter duplicate
detections. Moreover, we show performance improvements on nuScenes benchmark
dataset on top of our baselines.

提出了一种基于多视图轴自注意力和局部 ROI 自注意力的多路径多表示 3D 物体检测方法，通过稀疏浮动查询和密集 BEV 查询的多重表示进一步提高了性能，对于自动驾驶系统具有重要意义。