Vision-Language Pre-training (VLP) models trained on large-scale image-text
pairs have demonstrated unprecedented capability in many practical
applications. However, previous studies have revealed that VLP models are
vulnerable to adversarial samples crafted by a malicious adversary. While
existing attacks have achieved great success in improving attack effect and
transferability, they all focus on instance-specific attacks that generate
perturbations for each input sample. In this paper, we show that VLP models can
be vulnerable to a new class of universal adversarial perturbation (UAP) for
all input samples. Although initially transplanting existing UAP algorithms to
perform attacks showed effectiveness in attacking discriminative models, the
results were unsatisfactory when applied to VLP models. To this end, we revisit
the multimodal alignments in VLP model training and propose the
Contrastive-training Perturbation Generator with Cross-modal conditions
(C-PGC). Specifically, we first design a generator that incorporates
cross-modal information as conditioning input to guide the training. To further
exploit cross-modal interactions, we propose to formulate the training
objective as a multimodal contrastive learning paradigm based on our
constructed positive and negative image-text pairs. By training the conditional
generator with the designed loss, we successfully force the adversarial samples
to move away from its original area in the VLP model's feature space, and thus
essentially enhance the attacks. Extensive experiments show that our method
achieves remarkable attack performance across various VLP models and
Vision-and-Language (V+L) tasks. Moreover, C-PGC exhibits outstanding black-box
transferability and achieves impressive results in fooling prevalent large VLP
models including LLaVA and Qwen-VL.

我们展示了 Vision-Language Pre-training（VLP）模型对新型的通用对抗扰动（UAP）攻击的脆弱性，并提出了一种基于多模态对齐的 Contrastive-training Perturbation Generator 方法，该方法在各种 VLP 模型和多模态任务中展现出卓越的攻击性能，并表现出出色的黑盒可迁移性。