For web agents to be practically useful, they must adapt to the continuously
evolving web environment characterized by frequent updates to user interfaces
and content. However, most existing benchmarks only capture the static aspects
of the web. To bridge this gap, we introduce WebCanvas, an innovative online
evaluation framework for web agents that effectively addresses the dynamic
nature of web interactions. WebCanvas contains three main components to
facilitate realistic assessments: (1) A novel evaluation metric which reliably
capture critical intermediate actions or states necessary for task completions
while disregarding noise caused by insignificant events or changed
web-elements. (2) A benchmark dataset called Mind2Web-Live, a refined version
of original Mind2Web static dataset containing 542 tasks with 2439 intermediate
evaluation states; (3) Lightweight and generalizable annotation tools and
testing pipelines that enables the community to collect and maintain the
high-quality, up-to-date dataset. Building on WebCanvas, we open-source an
agent framework with extensible modules for reasoning, providing a foundation
for the community to conduct online inference and evaluations. Our
best-performing agent achieves a task success rate of 23.1% and a task
completion rate of 48.8% on the Mind2Web-Live test set. Additionally, we
analyze the performance discrepancies across various websites, domains, and
experimental environments. We encourage the community to contribute further
insights on online agent evaluation, thereby advancing this field of research.

WebCanvas 是一种创新的在线评估框架，用于有效解决 Web 交互的动态特性，包含评估指标、基准数据集和注释工具，并开源了可进行在线推理和评估的代理框架。

WebCanvas：在线环境下对网络代理进行基准测试

WebCanvas: Benchmarking Web Agents in Online Environments

With the fast development of large language models (LLMs), LLM-driven Web
Agents (Web Agents for short) have obtained tons of attention due to their
superior capability where LLMs serve as the core part of making decisions like
the human brain equipped with multiple web tools to actively interact with
external deployed websites. As uncountable Web Agents have been released and
such LLM systems are experiencing rapid development and drawing closer to
widespread deployment in our daily lives, an essential and pressing question
arises: "Are these Web Agents secure?". In this paper, we introduce a novel
threat, WIPI, that indirectly controls Web Agent to execute malicious
instructions embedded in publicly accessible webpages. To launch a successful
WIPI works in a black-box environment. This methodology focuses on the form and
content of indirect instructions within external webpages, enhancing the
efficiency and stealthiness of the attack. To evaluate the effectiveness of the
proposed methodology, we conducted extensive experiments using 7 plugin-based
ChatGPT Web Agents, 8 Web GPTs, and 3 different open-source Web Agents. The
results reveal that our methodology achieves an average attack success rate
(ASR) exceeding 90% even in pure black-box scenarios. Moreover, through an
ablation study examining various user prefix instructions, we demonstrated that
the WIPI exhibits strong robustness, maintaining high performance across
diverse prefix instructions.

LLM 驱动的 Web Agents（英语简写为 Web Agents）因其在像人脑一样通过多种网络工具与外部部署的网站进行积极互动时，具备强大的能力而备受关注。本文介绍了一种名为 WIPI 的新型威胁，其通过间接控制 Web Agents 在公开的网页中执行恶意指令。实验证明，即便在纯黑盒场景中，该方法达到了超过 90% 的平均攻击成功率（ASR），并且在多样化的前缀指令情况下仍具有高性能和强大的鲁棒性。

WIPI：基于 LLM 的网络代理新威胁

WIPI: A New Web Threat for LLM-Driven Web Agents

Web agents powered by Large Language Models (LLMs) have demonstrated
remarkable abilities in planning and executing multi-step interactions within
complex web-based environments, fulfilling a wide range of web navigation
tasks. Despite these advancements, the potential for LLM-powered agents to
effectively engage with sequential user instructions in real-world scenarios
has not been fully explored. In this work, we introduce a new task of
Conversational Web Navigation, which necessitates sophisticated interactions
that span multiple turns with both the users and the environment, supported by
a specially developed dataset named Multi-Turn Mind2Web (MT-Mind2Web). To
tackle the limited context length of LLMs and the context-dependency issue of
the conversational tasks, we further propose a novel framework, named
self-reflective memory-augmented planning (Self-MAP), which employs memory
utilization and self-reflection techniques. Extensive experiments are conducted
to benchmark the MT-Mind2Web dataset, and validate the effectiveness of the
proposed method.

本文介绍了一种名为 Conversational Web Navigation 的新任务，通过使用名为 Multi-Turn Mind2Web (MT-Mind2Web) 的特殊数据集，提出了一种名为 self-reflective memory-augmented planning (Self-MAP) 的新框架，通过利用记忆利用和自我反思技术，在现实世界的场景中有效地与连续的用户指令进行复杂的交互。

关于面向对话式网络代理的多轮指令遵循

On the Multi-turn Instruction Following for Conversational Web Agents

The advancement of large language models (LLMs) leads to a new era marked by
the development of autonomous applications in the real world, which drives
innovation in the creation of advanced web-based agents. Existing web agents
typically only handle one input modality and are evaluated only in simplified
web simulators or static web snapshots, greatly limiting their applicability in
real-world scenarios. To bridge this gap, we introduce WebVoyager, an
innovative Large Multimodal Model (LMM) powered web agent that can complete
user instructions end-to-end by interacting with real-world websites. Moreover,
we propose a new evaluation protocol for web agents to address the challenges
of automatic evaluation of open-ended web agent tasks, leveraging the robust
multimodal comprehension capabilities of GPT-4V. We create a new benchmark by
gathering real-world tasks from 15 widely used websites to evaluate our agents.
We show that WebVoyager achieves a 55.7% task success rate, significantly
surpassing the performance of both GPT-4 (All Tools) and the WebVoyager
(text-only) setups, underscoring the exceptional capability of WebVoyager in
practical applications. We found that our proposed automatic evaluation
achieves 85.3% agreement with human judgment, paving the way for further
development of web agents in a real-world setting.

我们介绍了 WebVoyager，这是一种创新的大型多模态模型（LMM）驱动的网络代理，可以通过与真实网站的交互来完成用户指令的端到端。此外，我们提出了一种新的网络代理评估协议，以解决开放式网络代理任务的自动评估挑战，利用了 GPT-4V 的强大多模态理解能力。我们通过收集 15 个广泛使用的网站上的真实任务来评估我们的代理，并展示了 WebVoyager 获得了 55.7% 的任务成功率，大大超过了 GPT-4（所有工具）和 WebVoyager（仅文本）设置的表现，突显出 WebVoyager 在实际应用中的卓越能力。我们发现我们提出的自动评估与人类判断达到了 85.3% 的一致性，为在真实世界中进一步开发网络代理铺平了道路。