For web agents to be practically useful, they must adapt to the continuously
evolving web environment characterized by frequent updates to user interfaces
and content. However, most existing benchmarks only capture the static aspects
of the web. To bridge this gap, we introduce WebCanvas, an innovative online
evaluation framework for web agents that effectively addresses the dynamic
nature of web interactions. WebCanvas contains three main components to
facilitate realistic assessments: (1) A novel evaluation metric which reliably
capture critical intermediate actions or states necessary for task completions
while disregarding noise caused by insignificant events or changed
web-elements. (2) A benchmark dataset called Mind2Web-Live, a refined version
of original Mind2Web static dataset containing 542 tasks with 2439 intermediate
evaluation states; (3) Lightweight and generalizable annotation tools and
testing pipelines that enables the community to collect and maintain the
high-quality, up-to-date dataset. Building on WebCanvas, we open-source an
agent framework with extensible modules for reasoning, providing a foundation
for the community to conduct online inference and evaluations. Our
best-performing agent achieves a task success rate of 23.1% and a task
completion rate of 48.8% on the Mind2Web-Live test set. Additionally, we
analyze the performance discrepancies across various websites, domains, and
experimental environments. We encourage the community to contribute further
insights on online agent evaluation, thereby advancing this field of research.

WebCanvas 是一种创新的在线评估框架，用于有效解决 Web 交互的动态特性，包含评估指标、基准数据集和注释工具，并开源了可进行在线推理和评估的代理框架。

WebCanvas：在线环境下对网络代理进行基准测试

WebCanvas: Benchmarking Web Agents in Online Environments

With generative AI advances, the exciting potential for autonomous agents to
manage daily tasks via natural language commands has emerged. However, cur rent
agents are primarily created and tested in simplified synthetic environments,
substantially limiting real-world scenario representation. In this paper, we
build an environment for agent command and control that is highly realistic and
reproducible. Specifically, we focus on agents that perform tasks on websites,
and we create an environment with fully functional websites from four common
domains: e-commerce, social forum discussions, collaborative software
development, and content management. Our environment is enriched with tools
(e.g., a map) and external knowledge bases (e.g., user manuals) to encourage
human-like task-solving. Building upon our environment, we release a set of
benchmark tasks focusing on evaluating the functional correctness of task
completions. The tasks in our benchmark are diverse, long-horizon, and are
designed to emulate tasks that humans routinely perform on the internet. We
design and implement several autonomous agents, integrating recent techniques
such as reasoning before acting. The results demonstrate that solving complex
tasks is challenging: our best GPT-4-based agent only achieves an end-to-end
task success rate of 10.59%. These results highlight the need for further
development of robust agents, that current state-of-the-art LMs are far from
perfect performance in these real-life tasks, and that WebArena can be used to
measure such progress. Our code, data, environment reproduction resources, and
video demonstrations are publicly available at this https URL

建立一个高度逼真和可重现的环境，专注于在网站上执行任务的智能代理，提供一组多样化、长期规划、模拟人类在互联网上例行执行的任务的基准任务以评估任务完成的功能正确性。