The recent development on large multimodal models (LMMs), especially
GPT-4V(ision) and Gemini, has been quickly expanding the capability boundaries
of multimodal models beyond traditional tasks like image captioning and visual
question answering. In this work, we explore the potential of LMMs like GPT-4V
as a generalist web agent that can follow natural language instructions to
complete tasks on any given website. We propose SEEACT, a generalist web agent
that harnesses the power of LMMs for integrated visual understanding and acting
on the web. We evaluate on the recent MIND2WEB benchmark. In addition to
standard offline evaluation on cached websites, we enable a new online
evaluation setting by developing a tool that allows running web agents on live
websites. We show that GPT-4V presents a great potential for web agents - it
can successfully complete 50% of the tasks on live websites if we manually
ground its textual plans into actions on the websites. This substantially
outperforms text-only LLMs like GPT-4 or smaller models (FLAN-T5 and BLIP-2)
specifically fine-tuned for web agents. However, grounding still remains a
major challenge. Existing LMM grounding strategies like set-of-mark prompting
turns out not effective for web agents, and the best grounding strategy we
develop in this paper leverages both the HTML text and visuals. Yet, there is
still a substantial gap with oracle grounding, leaving ample room for further
improvement.

本文研究了大型多模态模型 (LMMs)，特别是 GPT-4V (ision) 和 Gemini 的最新进展，扩展了多模态模型的能力边界，超出了传统任务如图像标题生成和视觉问答。我们提出了 SEEACT，一种利用 LMMs 的通用网络代理，可以根据自然语言指令在任何给定的网站上完成任务。通过在最新的 MIND2WEB 基准上进行评估，我们展示了 GPT-4V 在网络代理中的巨大潜力，如果我们将其文本计划手动转化为网站上的动作，它可以成功完成 50% 的任务，这大大优于仅针对网络代理进行优化的纯文本 LLMs，如 GPT-4 或更小的模型 (FLAN-T5 和 BLIP-2)。然而，转化仍然是一个主要的挑战，现有的 LMM 转化策略，如集合标记提示，并不适用于网络代理，而我们在本文中开发的最佳转化策略利用了 HTML 文本和视觉信息。然而，与理想转化仍然存在相当大的差距，这给进一步改进留下了充分的空间。