Bird's eye view (BEV) representation has emerged as a dominant solution for
describing 3D space in autonomous driving scenarios. However, objects in the
BEV representation typically exhibit small sizes, and the associated point
cloud context is inherently sparse, which leads to great challenges for
reliable 3D perception. In this paper, we propose IS-Fusion, an innovative
multimodal fusion framework that jointly captures the Instance- and Scene-level
contextual information. IS-Fusion essentially differs from existing approaches
that only focus on the BEV scene-level fusion by explicitly incorporating
instance-level multimodal information, thus facilitating the instance-centric
tasks like 3D object detection. It comprises a Hierarchical Scene Fusion (HSF)
module and an Instance-Guided Fusion (IGF) module. HSF applies Point-to-Grid
and Grid-to-Region transformers to capture the multimodal scene context at
different granularities. IGF mines instance candidates, explores their
relationships, and aggregates the local multimodal context for each instance.
These instances then serve as guidance to enhance the scene feature and yield
an instance-aware BEV representation. On the challenging nuScenes benchmark,
IS-Fusion outperforms all the published multimodal works to date. Code is
available at: this https URL

本文提出了一种创新的多模态融合框架 IS-Fusion，旨在解决自动驾驶场景中描述三维空间所面临的挑战。通过捕捉实例和场景级别的上下文信息，IS-Fusion 在处理 3D 物体检测等以实例为中心的任务时具有明显优势，超过了现有的多模态方法。

IS-Fusion: 多模态三维物体检测的实例场景协同融合

IS-Fusion: Instance-Scene Collaborative Fusion for Multimodal 3D Object  Detection

There are increasing concerns about malicious attacks on autonomous vehicles.
In particular, inaudible voice command attacks pose a significant threat as
voice commands become available in autonomous driving systems. How to
empirically defend against these inaudible attacks remains an open question.
Previous research investigates utilizing deep learning-based multimodal fusion
for defense, without considering the model uncertainty in trustworthiness. As
deep learning has been applied to increasingly sensitive tasks, uncertainty
measurement is crucial in helping improve model robustness, especially in
mission-critical scenarios. In this paper, we propose the Multimodal Fusion
Framework (MFF) as an intelligent security system to defend against inaudible
voice command attacks. MFF fuses heterogeneous audio-vision modalities using
VGG family neural networks and achieves the detection accuracy of 92.25% in the
comparative fusion method empirical study. Additionally, extensive experiments
on audio-vision tasks reveal the model's uncertainty. Using Expected
Calibration Errors, we measure calibration errors and Monte-Carlo Dropout to
estimate the predictive distribution for the proposed models. Our findings show
empirically to train robust multimodal models, improve standard accuracy and
provide a further step toward interpretability. Finally, we discuss the pros
and cons of our approach and its applicability for Advanced Driver Assistance
Systems.

本文提出了一种多模态融合 (Multimodal Fusion Framework) 安全系统，利用 VGG 系列神经网络融合异构音频 - 视觉模态，检测准确率可达 92.25%, 并测量了模型不确定性，结果表明我们的方法可帮助提高模型鲁棒性和准确性，并为理解提供了更深入的步骤，适用于高级驾驶辅助系统。

高级驾驶辅助系统中抵御无声指令攻击的可信传感器融合方案

Trustworthy Sensor Fusion against Inaudible Command Attacks in Advanced  Driver-Assistance System

Deep neural networks (DNNs) provide state-of-the-art results for a multitude
of applications, but the approaches using DNNs for multimodal audiovisual
applications do not consider predictive uncertainty associated with individual
modalities. Bayesian deep learning methods provide principled confidence and
quantify predictive uncertainty. Our contribution in this work is to propose an
uncertainty aware multimodal Bayesian fusion framework for activity
recognition. We demonstrate a novel approach that combines deterministic and
variational layers to scale Bayesian DNNs to deeper architectures. Our
experiments using in- and out-of-distribution samples selected from a subset of
Moments-in-Time (MiT) dataset show a more reliable confidence measure as
compared to the non-Bayesian baseline and the Monte Carlo dropout (MC dropout)
approximate Bayesian inference. We also demonstrate the uncertainty estimates
obtained from the proposed framework can identify out-of-distribution data on
the UCF101 and MiT datasets. In the multimodal setting, the proposed framework
improved precision-recall AUC by 10.2% on the subset of MiT dataset as compared
to non-Bayesian baseline.

本研究提出了一种考虑个体单模态预测不确定性的多模态贝叶斯融合框架，采用行为识别作为应用案例，并结合确定性和变分层混合以扩展贝叶斯深度神经网络的深层架构，在 Moments-in-Time 数据集的实验中表现出优于基准方法和 Monte Carlo dropout 方法的可靠置信度，并在 UCF101 和 MiT 数据集上证明了该框架从中提取的预测不确定性能够识别元数据。