Large Language Models (LLMs) are being enhanced with the ability to use tools
and to process multiple modalities. These new capabilities bring new benefits
and also new security risks. In this work, we show that an attacker can use
visual adversarial examples to cause attacker-desired tool usage. For example,
the attacker could cause a victim LLM to delete calendar events, leak private
conversations and book hotels. Different from prior work, our attacks can
affect the confidentiality and integrity of user resources connected to the LLM
while being stealthy and generalizable to multiple input prompts. We construct
these attacks using gradient-based adversarial training and characterize
performance along multiple dimensions. We find that our adversarial images can
manipulate the LLM to invoke tools following real-world syntax almost always
(~98%) while maintaining high similarity to clean images (~0.9 SSIM).
Furthermore, using human scoring and automated metrics, we find that the
attacks do not noticeably affect the conversation (and its semantics) between
the user and the LLM.

大型语言模型对多模态处理和工具使用能力的增强带来了新的好处和安全风险，本文揭示了攻击者可以使用视觉对抗样本来引发特定工具使用的能力。攻击可以影响与 LLM 连接的用户资源的机密性和完整性，同时保持隐秘性，并且可泛化到多个输入提示。通过基于梯度的对抗训练构建这些攻击，并在多个维度上表征其性能。研究发现，我们的对抗图像几乎总能以接近真实语法的方式操控 LLM 来调用工具（准确率约为 98%），同时保持与原始图像的高相似度（约 0.9 SSIM）。此外，通过人工评分和自动化指标，我们发现这些攻击对用户与 LLM 之间的对话（及其语义）没有显著影响。

在大型语言模型中使用视觉对抗样本错误使用工具

Misusing Tools in Large Language Models With Visual Adversarial Examples

Visual adversarial examples have so far been restricted to pixel-level image
manipulations in the digital world, or have required sophisticated equipment
such as 2D or 3D printers to be produced in the physical real world. We present
the first ever method of generating human-producible adversarial examples for
the real world that requires nothing more complicated than a marker pen. We
call them $\textbf{adversarial tags}$. First, building on top of differential
rendering, we demonstrate that it is possible to build potent adversarial
examples with just lines. We find that by drawing just $4$ lines we can disrupt
a YOLO-based model in $54.8\%$ of cases; increasing this to $9$ lines disrupts
$81.8\%$ of the cases tested. Next, we devise an improved method for line
placement to be invariant to human drawing error. We evaluate our system
thoroughly in both digital and analogue worlds and demonstrate that our tags
can be applied by untrained humans. We demonstrate the effectiveness of our
method for producing real-world adversarial examples by conducting a user study
where participants were asked to draw over printed images using digital
equivalents as guides. We further evaluate the effectiveness of both targeted
and untargeted attacks, and discuss various trade-offs and method limitations,
as well as the practical and ethical implications of our work. The source code
will be released publicly.

用区别性渲染的方法，我们首次提出了一种生成真实世界中可由人们创造的对抗性样本的方法，只需要一支马克笔。我们通过绘制少量线条便可以干扰 YOLO 模型，并通过数字和模拟环境进行全面评估，展示了我们的方法的有效性。

可人生成的敌制式示例

Human-Producible Adversarial Examples

While text-based machine learning models that operate on visual inputs of
rendered text have become robust against a wide range of existing attacks, we
show that they are still vulnerable to visual adversarial examples encoded as
text. We use the Unicode functionality of combining diacritical marks to
manipulate encoded text so that small visual perturbations appear when the text
is rendered. We show how a genetic algorithm can be used to generate visual
adversarial examples in a black-box setting, and conduct a user study to
establish that the model-fooling adversarial examples do not affect human
comprehension. We demonstrate the effectiveness of these attacks in the real
world by creating adversarial examples against production models published by
Facebook, Microsoft, IBM, and Google.

本文探讨了基于文字的机器学习模型对视觉输入的抵抗力，并表明他们仍然容易受到以文本编码的视觉对抗性示例的攻击，使用 Unicode 功能组合变音符号来操纵编码文本，演示使用遗传算法生成黑盒威胁视觉对抗性示例的效力，并进行用户研究以证明这些模型欺骗性示例不会影响人类理解。最后，通过制造针对 Facebook，Microsoft，IBM 和 Google 发布的生产模型的对抗性示例，展示了这些攻击的实际效果。