Vision-enabled language models (VLMs) are now used to build autonomous
multimodal agents capable of taking actions in real environments. In this
paper, we show that multimodal agents raise new safety risks, even though
attacking agents is more challenging than prior attacks due to limited access
to and knowledge about the environment. Our attacks use adversarial text
strings to guide gradient-based perturbation over one trigger image in the
environment: (1) our captioner attack attacks white-box captioners if they are
used to process images into captions as additional inputs to the VLM; (2) our
CLIP attack attacks a set of CLIP models jointly, which can transfer to
proprietary VLMs. To evaluate the attacks, we curated VisualWebArena-Adv, a set
of adversarial tasks based on VisualWebArena, an environment for web-based
multimodal agent tasks. Within an L-infinity norm of $16/256$ on a single
image, the captioner attack can make a captioner-augmented GPT-4V agent execute
the adversarial goals with a 75% success rate. When we remove the captioner or
use GPT-4V to generate its own captions, the CLIP attack can achieve success
rates of 21% and 43%, respectively. Experiments on agents based on other VLMs,
such as Gemini-1.5, Claude-3, and GPT-4o, show interesting differences in their
robustness. Further analysis reveals several key factors contributing to the
attack's success, and we also discuss the implications for defenses as well.
Project page: this https URL Code and data:
this https URL

基于视觉和语言的模型对自主多模态代理进行了攻击，展示了新的安全风险，并通过对抗性文本字符串指导基于梯度的扰动来执行对手目标。

多模态代理的对抗攻击

Adversarial Attacks on Multimodal Agents

Autonomous agents that accomplish complex computer tasks with minimal human
interventions have the potential to transform human-computer interaction,
significantly enhancing accessibility and productivity. However, existing
benchmarks either lack an interactive environment or are limited to
environments specific to certain applications or domains, failing to reflect
the diverse and complex nature of real-world computer use, thereby limiting the
scope of tasks and agent scalability. To address this issue, we introduce
OSWorld, the first-of-its-kind scalable, real computer environment for
multimodal agents, supporting task setup, execution-based evaluation, and
interactive learning across various operating systems such as Ubuntu, Windows,
and macOS. OSWorld can serve as a unified, integrated computer environment for
assessing open-ended computer tasks that involve arbitrary applications.
Building upon OSWorld, we create a benchmark of 369 computer tasks involving
real web and desktop apps in open domains, OS file I/O, and workflows spanning
multiple applications. Each task example is derived from real-world computer
use cases and includes a detailed initial state setup configuration and a
custom execution-based evaluation script for reliable, reproducible evaluation.
Extensive evaluation of state-of-the-art LLM/VLM-based agents on OSWorld
reveals significant deficiencies in their ability to serve as computer
assistants. While humans can accomplish over 72.36% of the tasks, the best
model achieves only 12.24% success, primarily struggling with GUI grounding and
operational knowledge. Comprehensive analysis using OSWorld provides valuable
insights for developing multimodal generalist agents that were not possible
with previous benchmarks. Our code, environment, baseline models, and data are
publicly available at this https URL

通过引入 OSWorld，我们创建了一个包含 369 个计算机任务的基准，以评估多模态代理在开放领域中执行任意应用程序所涉及的计算机任务的能力。在 OSWorld 上进行的全面评估为开发多模态通用代理提供了宝贵的洞见，这是以前的基准测试无法实现的。

OSWorld: 多模态代理在真实计算机环境中的开放式任务评测

OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real  Computer Environments

Large language models (LLMs) have achieved superior performance in powering
text-based AI agents, endowing them with decision-making and reasoning
abilities akin to humans. Concurrently, there is an emerging research trend
focused on extending these LLM-powered AI agents into the multimodal domain.
This extension enables AI agents to interpret and respond to diverse multimodal
user queries, thereby handling more intricate and nuanced tasks. In this paper,
we conduct a systematic review of LLM-driven multimodal agents, which we refer
to as large multimodal agents ( LMAs for short). First, we introduce the
essential components involved in developing LMAs and categorize the current
body of research into four distinct types. Subsequently, we review the
collaborative frameworks integrating multiple LMAs , enhancing collective
efficacy. One of the critical challenges in this field is the diverse
evaluation methods used across existing studies, hindering effective comparison
among different LMAs . Therefore, we compile these evaluation methodologies and
establish a comprehensive framework to bridge the gaps. This framework aims to
standardize evaluations, facilitating more meaningful comparisons. Concluding
our review, we highlight the extensive applications of LMAs and propose
possible future research directions. Our discussion aims to provide valuable
insights and guidelines for future research in this rapidly evolving field. An
up-to-date resource list is available at
this https URL

大型语言模型在多媒体领域的研究和发展方向的全面回顾，介绍了大型多媒体代理的开发要素和分类，评估方法的编制以及未来研究方向的提出。