Vision-Language (VL) pre-trained models have shown their superiority on many
multimodal tasks. However, the adversarial robustness of such models has not
been fully explored. Existing approaches mainly focus on exploring the
adversarial robustness under the white-box setting, which is unrealistic. In
this paper, we aim to investigate a new yet practical task to craft image and
text perturbations using pre-trained VL models to attack black-box fine-tuned
models on different downstream tasks. Towards this end, we propose VLAttack to
generate adversarial samples by fusing perturbations of images and texts from
both single-modal and multimodal levels. At the single-modal level, we propose
a new block-wise similarity attack (BSA) strategy to learn image perturbations
for disrupting universal representations. Besides, we adopt an existing text
attack strategy to generate text perturbations independent of the image-modal
attack. At the multimodal level, we design a novel iterative cross-search
attack (ICSA) method to update adversarial image-text pairs periodically,
starting with the outputs from the single-modal level. We conduct extensive
experiments to attack three widely-used VL pretrained models for six tasks on
eight datasets. Experimental results show that the proposed VLAttack framework
achieves the highest attack success rates on all tasks compared with
state-of-the-art baselines, which reveals a significant blind spot in the
deployment of pre-trained VL models. Codes will be released soon.

本文研究了使用图像和文本扰动生成对黑盒微调模型进行攻击的新的实用任务，提出了 VLAttack 框架，通过融合单模态和多模态层次的图像和文本扰动来生成对抗样本，实验结果表明该框架在各项任务上攻击成功率最高，揭示了预训练 Vision-Language 模型部署中的一个重要盲点。