The success of deep learning has enabled advances in multimodal tasks that
require non-trivial fusion of multiple input domains. Although multimodal
models have shown potential in many problems, their increased complexity makes
them more vulnerable to attacks. A Backdoor (or Trojan) attack is a class of
security vulnerability wherein an attacker embeds a malicious secret behavior
into a network (e.g. targeted misclassification) that is activated when an
attacker-specified trigger is added to an input. In this work, we show that
multimodal networks are vulnerable to a novel type of attack that we refer to
as Dual-Key Multimodal Backdoors. This attack exploits the complex fusion
mechanisms used by state-of-the-art networks to embed backdoors that are both
effective and stealthy. Instead of using a single trigger, the proposed attack
embeds a trigger in each of the input modalities and activates the malicious
behavior only when both the triggers are present. We present an extensive study
of multimodal backdoors on the Visual Question Answering (VQA) task with
multiple architectures and visual feature backbones. A major challenge in
embedding backdoors in VQA models is that most models use visual features
extracted from a fixed pretrained object detector. This is challenging for the
attacker as the detector can distort or ignore the visual trigger entirely,
which leads to models where backdoors are over-reliant on the language trigger.
We tackle this problem by proposing a visual trigger optimization strategy
designed for pretrained object detectors. Through this method, we create
Dual-Key Backdoors with over a 98% attack success rate while only poisoning 1%
of the training data. Finally, we release TrojVQA, a large collection of clean
and trojan VQA models to enable research in defending against multimodal
backdoors.

研究表明，多模态网络容易受到后门攻击，提出了一种称为双重密钥多模态后门攻击的新型攻击，通过视觉触发器优化策略，攻击成功率高达 98％，仅污染 1％的训练数据，最终发布了 TrojVQA，一个大型的干净和 Trojan VQA 模型集合，用于研究防御多模态后门攻击。