The progress of autonomous web navigation has been hindered by the dependence
on billions of exploratory interactions via online reinforcement learning, and
domain-specific model designs that make it difficult to leverage generalization
from rich out-of-domain data. In this work, we study data-driven offline
training for web agents with vision-language foundation models. We propose an
instruction-following multimodal agent, WebGUM, that observes both webpage
screenshots and HTML pages and outputs web navigation actions, such as click
and type. WebGUM is trained by jointly finetuning an instruction-finetuned
language model and a vision transformer on a large corpus of demonstrations. We
empirically demonstrate this recipe improves the agent's ability of grounded
visual perception, HTML comprehension and multi-step reasoning, outperforming
prior works by a significant margin. On the MiniWoB benchmark, we improve over
the previous best offline methods by more than 31.9%, being close to reaching
online-finetuned SoTA. On the WebShop benchmark, our 3-billion-parameter model
achieves superior performance to the existing SoTA, PaLM-540B. We also collect
347K high-quality demonstrations using our trained models, 38 times larger than
prior work, and make them available to promote future research in this
direction.

本文提出了一种基于视觉 - 语言模型的指令驱动表征，通过离线培训方式对 WebGUM 模型进行了训练，使其在视觉感知、HTML 理解和多步推理能力上表现出色，相比现有最佳方法提高了 31.9％以上。