In this paper, we propose to make a systematic study on machines multisensory
perception under attacks. We use the audio-visual event recognition task
against multimodal adversarial attacks as a proxy to investigate the robustness
of audio-visual learning. We attack audio, visual, and both modalities to
explore whether audio-visual integration still strengthens perception and how
different fusion mechanisms affect the robustness of audio-visual models. For
interpreting the multimodal interactions under attacks, we learn a
weakly-supervised sound source visual localization model to localize sounding
regions in videos. To mitigate multimodal attacks, we propose an audio-visual
defense approach based on an audio-visual dissimilarity constraint and external
feature memory banks. Extensive experiments demonstrate that audio-visual
models are susceptible to multimodal adversarial attacks; audio-visual
integration could decrease the model robustness rather than strengthen under
multimodal attacks; even a weakly-supervised sound source visual localization
model can be successfully fooled; our defense method can improve the
invulnerability of audio-visual networks without significantly sacrificing
clean model performance.

该论文旨在研究多模态攻击下机器的多感官感知能力，针对多模态对抗攻击对视听事件识别任务的影响，提出弱监督声音定位模型并基于音视频相似性约束和外部特征存储器实现了一种反对抗防御方法，实验结果表明该方法可有效提高视听网络的鲁棒性。