Beyond achieving high performance across many vision tasks, multimodal models
are expected to be robust to single-source faults due to the availability of
redundant information between modalities. In this paper, we investigate the
robustness of multimodal neural networks against worst-case (i.e., adversarial)
perturbations on a single modality. We first show that standard multimodal
fusion models are vulnerable to single-source adversaries: an attack on any
single modality can overcome the correct information from multiple unperturbed
modalities and cause the model to fail. This surprising vulnerability holds
across diverse multimodal tasks and necessitates a solution. Motivated by this
finding, we propose an adversarially robust fusion strategy that trains the
model to compare information coming from all the input sources, detect
inconsistencies in the perturbed modality compared to the other modalities, and
only allow information from the unperturbed modalities to pass through. Our
approach significantly improves on state-of-the-art methods in single-source
robustness, achieving gains of 7.8-25.2% on action recognition, 19.7-48.2% on
object detection, and 1.6-6.7% on sentiment analysis, without degrading
performance on unperturbed (i.e., clean) data.

本研究旨在分析多模态神经网络在单模态最坏情况下 (即对抗性攻击) 的鲁棒性，并提出了一种对抗性融合策略，该策略与现有方法相比，可显著提高单源鲁棒性，并在不影响干净数据性能的情况下，在多种多模态任务上实现了良好的表现。