Multi-modal Large Language Models (MLLMs) have recently achieved enhanced
performance across various vision-language tasks including visual grounding
capabilities. However, the adversarial robustness of visual grounding remains
unexplored in MLLMs. To fill this gap, we use referring expression
comprehension (REC) as an example task in visual grounding and propose three
adversarial attack paradigms as follows. Firstly, untargeted adversarial
attacks induce MLLMs to generate incorrect bounding boxes for each object.
Besides, exclusive targeted adversarial attacks cause all generated outputs to
the same target bounding box. In addition, permuted targeted adversarial
attacks aim to permute all bounding boxes among different objects within a
single image. Extensive experiments demonstrate that the proposed methods can
successfully attack visual grounding capabilities of MLLMs. Our methods not
only provide a new perspective for designing novel attacks but also serve as a
strong baseline for improving the adversarial robustness for visual grounding
of MLLMs.

使用多模态大型语言模型（MLLMs）对视觉定位进行饱受关注的对抗性攻击研究，通过提出三种对抗攻击范式来攻击 MLLMs 的视觉定位性能，进而为提高 MLLMs 的视觉定位的对抗鲁棒性提供了新的视角和强大的基线。