Multi-sensor modal fusion has demonstrated strong advantages in 3D object
detection tasks. However, existing methods that fuse multi-modal features
through a simple channel concatenation require transformation features into
bird's eye view space and may lose the information on Z-axis thus leads to
inferior performance. To this end, we propose FusionFormer, an end-to-end
multi-modal fusion framework that leverages transformers to fuse multi-modal
features and obtain fused BEV features. And based on the flexible adaptability
of FusionFormer to the input modality representation, we propose a depth
prediction branch that can be added to the framework to improve detection
performance in camera-based detection tasks. In addition, we propose a
plug-and-play temporal fusion module based on transformers that can fuse
historical frame BEV features for more stable and reliable detection results.
We evaluate our method on the nuScenes dataset and achieve 72.6% mAP and 75.1%
NDS for 3D object detection tasks, outperforming state-of-the-art methods.

多传感器模态融合的 FusionFormer 框架通过 transformers 混合多模态特征，获取鸟瞰图特征，并通过可灵活适应输入模态表示的深度预测分支提高相机检测任务的性能，同时提出一种基于 transformers 的即插即用的时域融合模块用于融合历史帧鸟瞰图特征，实现更稳定可靠的检测结果，通过对 nuScenes 数据集的评估，其在 3D 目标检测任务中获得了 72.6％的 mAP 和 75.1％的 NDS，优于现有的最先进方法。

FusionFormer：鸟瞰视图和时间一致 Transformer 的多感知融合用于 3D 目标

FusionFormer: A Multi-sensory Fusion in Bird's-Eye-View and Temporal  Consistent Transformer for 3D Objection

The recently proposed Conformer architecture which combines convolution with
attention to capture both local and global dependencies has become the
\textit{de facto} backbone model for Automatic Speech Recognition~(ASR).
Inherited from the Natural Language Processing (NLP) tasks, the architecture
takes Layer Normalization~(LN) as a default normalization technique. However,
through a series of systematic studies, we find that LN might take 10\% of the
inference time despite that it only contributes to 0.1\% of the FLOPs. This
motivates us to replace LN with other normalization techniques, e.g., Batch
Normalization~(BN), to speed up inference with the help of operator fusion
methods and the avoidance of calculating the mean and variance statistics
during inference. After examining several plain attempts which directly remove
all LN layers or replace them with BN in the same place, we find that the
divergence issue is mainly caused by the unstable layer output. We therefore
propose to append a BN layer to each linear or convolution layer where
stabilized training results are observed. We also propose to simplify the
activations in Conformer, such as Swish and GLU, by replacing them with ReLU.
All these exchanged modules can be fused into the weights of the adjacent
linear/convolution layers and hence have zero inference cost. Therefore, we
name it FusionFormer. Our experiments indicate that FusionFormer is as
effective as the LN-based Conformer and is about 10\% faster.

通过将层标准化技术替换为批标准化技术并且对 Conformer 架构的激活函数进行简化和替换（用 ReLU），我们提出了一个名为 FusionFormer 的模型，它的推理速度比标准的 Conformer 模型快约 10%，而且效果相当。