The progress of autonomous web navigation has been hindered by the dependence
on billions of exploratory interactions via online reinforcement learning, and
domain-specific model designs that make it difficult to leverage generalization
from rich out-of-domain data. In this work, we study data-driven offline
training for web agents with vision-language foundation models. We propose an
instruction-following multimodal agent, WebGUM, that observes both webpage
screenshots and HTML pages and outputs web navigation actions, such as click
and type. WebGUM is trained by jointly finetuning an instruction-finetuned
language model and a vision transformer on a large corpus of demonstrations. We
empirically demonstrate this recipe improves the agent's ability of grounded
visual perception, HTML comprehension and multi-step reasoning, outperforming
prior works by a significant margin. On the MiniWoB benchmark, we improve over
the previous best offline methods by more than 31.9%, being close to reaching
online-finetuned SoTA. On the WebShop benchmark, our 3-billion-parameter model
achieves superior performance to the existing SoTA, PaLM-540B. We also collect
347K high-quality demonstrations using our trained models, 38 times larger than
prior work, and make them available to promote future research in this
direction.

本文提出了一种基于视觉 - 语言模型的指令驱动表征，通过离线培训方式对 WebGUM 模型进行了训练，使其在视觉感知、HTML 理解和多步推理能力上表现出色，相比现有最佳方法提高了 31.9％以上。

基于指令微调基础模型的多模式网络导航

Multimodal Web Navigation with Instruction-Finetuned Foundation Models

Large language models (LLMs) have shown exceptional performance on a variety
of natural language tasks. Yet, their capabilities for HTML understanding --
i.e., parsing the raw HTML of a webpage, with applications to automation of
web-based tasks, crawling, and browser-assisted retrieval -- have not been
fully explored. We contribute HTML understanding models (fine-tuned LLMs) and
an in-depth analysis of their capabilities under three tasks: (i) Semantic
Classification of HTML elements, (ii) Description Generation for HTML inputs,
and (iii) Autonomous Web Navigation of HTML pages. While previous work has
developed dedicated architectures and training procedures for HTML
understanding, we show that LLMs pretrained on standard natural language
corpora transfer remarkably well to HTML understanding tasks. For instance,
fine-tuned LLMs are 12% more accurate at semantic classification compared to
models trained exclusively on the task dataset. Moreover, when fine-tuned on
data from the MiniWoB benchmark, LLMs successfully complete 50% more tasks
using 192x less data compared to the previous best supervised model. Out of the
LLMs we evaluate, we show evidence that T5-based models are ideal due to their
bidirectional encoder-decoder architecture. To promote further research on LLMs
for HTML understanding, we create and open-source a large-scale HTML dataset
distilled and auto-labeled from CommonCrawl.

通过 fine-tuned LLMs 调整，研究 HTML 理解（元素语义分类，输入描述生成和自主网页导航）任务。 T5-based models 表现最佳，提出一个来自 CommonCrawl 的大规模 HTML 数据集。