Autonomous agents capable of planning, reasoning, and executing actions on
the web offer a promising avenue for automating computer tasks. However, the
majority of existing benchmarks primarily focus on text-based agents,
neglecting many natural tasks that require visual information to effectively
solve. Given that most computer interfaces cater to human perception, visual
information often augments textual data in ways that text-only models struggle
to harness effectively. To bridge this gap, we introduce VisualWebArena, a
benchmark designed to assess the performance of multimodal web agents on
realistic \textit{visually grounded tasks}. VisualWebArena comprises of a set
of diverse and complex web-based tasks that evaluate various capabilities of
autonomous multimodal agents. To perform on this benchmark, agents need to
accurately process image-text inputs, interpret natural language instructions,
and execute actions on websites to accomplish user-defined objectives. We
conduct an extensive evaluation of state-of-the-art LLM-based autonomous
agents, including several multimodal models. Through extensive quantitative and
qualitative analysis, we identify several limitations of text-only LLM agents,
and reveal gaps in the capabilities of state-of-the-art multimodal language
agents. VisualWebArena provides a framework for evaluating multimodal
autonomous language agents, and offers insights towards building stronger
autonomous agents for the web. Our code, baseline models, and data is publicly
available at this https URL

通过对多模态网络代理的性能进行评估，我们引入了 VisualWebArena，它是一个用于评估自主多模态代理在具有视觉基础任务方面性能的基准。我们对现有的自主代理进行了广泛的评估，并揭示了文本模型的几个限制以及现有多模态语言代理能力上的差距。