Diffusion models have recently gained prominence as powerful deep generative
models, demonstrating unmatched performance across various domains. However,
their potential in multi-sensor fusion remains largely unexplored. In this
work, we introduce DifFUSER, a novel approach that leverages diffusion models
for multi-modal fusion in 3D object detection and BEV map segmentation.
Benefiting from the inherent denoising property of diffusion, DifFUSER is able
to refine or even synthesize sensor features in case of sensor malfunction,
thereby improving the quality of the fused output. In terms of architecture,
our DifFUSER blocks are chained together in a hierarchical BiFPN fashion,
termed cMini-BiFPN, offering an alternative architecture for latent diffusion.
We further introduce a Gated Self-conditioned Modulated (GSM) latent diffusion
module together with a Progressive Sensor Dropout Training (PSDT) paradigm,
designed to add stronger conditioning to the diffusion process and robustness
to sensor failures. Our extensive evaluations on the Nuscenes dataset reveal
that DifFUSER not only achieves state-of-the-art performance with a 69.1% mIOU
in BEV map segmentation tasks but also competes effectively with leading
transformer-based fusion techniques in 3D object detection.

DifFUSER 是一个新的方法，利用扩散模型进行多模态融合，特别适用于三维物体检测和 BEV 地图分割，通过扩散的去噪特性，能够在传感器故障的情况下改进或合成传感器特征，从而提高融合输出的质量。在架构方面，我们采用了层级 BiFPN 结构的 DifFUSER 块，进一步引入了 Gated Self-conditioned Modulated (GSM) 潜在扩散模块以及 Progressive Sensor Dropout Training (PSDT) 范式，以增强扩散过程的条件性和对传感器故障的鲁棒性。我们在 Nuscenes 数据集上进行广泛评估，结果显示 DifFUSER 不仅在 BEV 地图分割任务中取得了 69.1% 的 mIOU 的最新成绩，而且在三维物体检测方面与领先的基于 Transformer 的融合技术具有很强的竞争力。

DifFUSER: 三维物体检测和 BEV 分割中鲁棒多传感器融合的扩散模型

DifFUSER: Diffusion Model for Robust Multi-Sensor Fusion in 3D Object  Detection and BEV Segmentation

Accurate and comprehensive semantic segmentation of Bird's Eye View (BEV) is
essential for ensuring safe and proactive navigation in autonomous driving.
Although cooperative perception has exceeded the detection capabilities of
single-agent systems, prevalent camera-based algorithms in cooperative
perception neglect valuable information derived from historical observations.
This limitation becomes critical during sensor failures or communication issues
as cooperative perception reverts to single-agent perception, leading to
degraded performance and incomplete BEV segmentation maps. This paper
introduces TempCoBEV, a temporal module designed to incorporate historical cues
into current observations, thereby improving the quality and reliability of BEV
map segmentations. We propose an importance-guided attention architecture to
effectively integrate temporal information that prioritizes relevant properties
for BEV map segmentation. TempCoBEV is an independent temporal module that
seamlessly integrates into state-of-the-art camera-based cooperative perception
models. We demonstrate through extensive experiments on the OPV2V dataset that
TempCoBEV performs better than non-temporal models in predicting current and
future BEV map segmentations, particularly in scenarios involving communication
failures. We show the efficacy of TempCoBEV and its capability to integrate
historical cues into the current BEV map, improving predictions under optimal
communication conditions by up to 2% and under communication failures by up to
19%. The code will be published on GitHub.

通过将历史线索整合到当前 BEV 地图中，TempCoBEV 能够在协作感知中改进 BEV 地图分割的质量和可靠性。在 OPV2V 数据集上进行的广泛实验表明，TempCoBEV 在预测当前和未来的 BEV 地图分割方面比非时态模型表现更好，特别是在涉及通信故障的情景中，其整合历史线索到当前 BEV 地图的能力使预测结果在最佳通信条件下提高 2%，在通信故障下提高 19%。