In typical multimodal tasks, such as Visual Question Answering (VQA), adversarial attacks targeting a specific image and question can lead large vision-language models (LVLMs) to provide incorrect answers. However, it is common for a single image to be associated with multiple questions, and LVLMs may still answer other questions correctly even for an adversarial image attacked by a specific question. To address this, we introduce the query-agnostic visual attack (QAVA), which aims to create robust adversarial examples that generate incorrect responses to unspecified and unknown questions. Compared to traditional adversarial attacks focused on specific images and questions, QAVA significantly enhances the effectiveness and efficiency of attacks on images when the question is unknown, achieving performance comparable to attacks on known target questions. Our research broadens the scope of visual adversarial attacks on LVLMs in practical settings, uncovering previously overlooked vulnerabilities, particularly in the context of visual adversarial threats. The code is available at https://github.com/btzyd/qava.

本研究针对大规模视觉语言模型（LVLMs）在视觉问答（VQA）任务中对特定图像和问题的脆弱性，提出了一种查询无关视觉攻击（QAVA）。这一新方法能够生成对未知问题产生错误响应的稳健对抗样本，从而显著提高了在未知问题情况下攻击的有效性和效率，揭示了LVLMs在视觉对抗威胁中的新兴漏洞。

QAVA：针对大型视觉语言模型的查询无关视觉攻击