Web agents powered by Large Language Models (LLMs) have demonstrated
remarkable abilities in planning and executing multi-step interactions within
complex web-based environments, fulfilling a wide range of web navigation
tasks. Despite these advancements, the potential for LLM-powered agents to
effectively engage with sequential user instructions in real-world scenarios
has not been fully explored. In this work, we introduce a new task of
Conversational Web Navigation, which necessitates sophisticated interactions
that span multiple turns with both the users and the environment, supported by
a specially developed dataset named Multi-Turn Mind2Web (MT-Mind2Web). To
tackle the limited context length of LLMs and the context-dependency issue of
the conversational tasks, we further propose a novel framework, named
self-reflective memory-augmented planning (Self-MAP), which employs memory
utilization and self-reflection techniques. Extensive experiments are conducted
to benchmark the MT-Mind2Web dataset, and validate the effectiveness of the
proposed method.

本文介绍了一种名为 Conversational Web Navigation 的新任务，通过使用名为 Multi-Turn Mind2Web (MT-Mind2Web) 的特殊数据集，提出了一种名为 self-reflective memory-augmented planning (Self-MAP) 的新框架，通过利用记忆利用和自我反思技术，在现实世界的场景中有效地与连续的用户指令进行复杂的交互。

关于面向对话式网络代理的多轮指令遵循

On the Multi-turn Instruction Following for Conversational Web Agents

We propose the problem of conversational web navigation, where a digital
agent controls a web browser and follows user instructions to solve real-world
tasks in a multi-turn dialogue fashion. To support this problem, we introduce
WEBLINX - a large-scale benchmark of 100K interactions across 2300 expert
demonstrations of conversational web navigation. Our benchmark covers a broad
range of patterns on over 150 real-world websites and can be used to train and
evaluate agents in diverse scenarios. Due to the magnitude of information
present, Large Language Models (LLMs) cannot process entire web pages in
real-time. To solve this bottleneck, we design a retrieval-inspired model that
efficiently prunes HTML pages by ranking relevant elements. We use the selected
elements, along with screenshots and action history, to assess a variety of
models for their ability to replicate human behavior when navigating the web.
Our experiments span from small text-only to proprietary multimodal LLMs. We
find that smaller finetuned decoders surpass the best zero-shot LLMs (including
GPT-4V), but also larger finetuned multimodal models which were explicitly
pretrained on screenshots. However, all finetuned models struggle to generalize
to unseen websites. Our findings highlight the need for large multimodal models
that can generalize to novel settings. Our code, data and models are available
for research: this https URL

我们提出了会话式网络导航的问题，其中数字代理控制网络浏览器，并遵循用户的指令以多轮对话的方式解决现实任务。为了支持这个问题，我们介绍了 WEBLINX - 一个包括多个领域的 100K 个交互和 2300 个专家演示的大规模基准。我们的基准涵盖了 150 多个真实网站上的各种模式，并可用于在不同场景中训练和评估代理。由于存在大量信息，大型语言模型（LLM）无法实时处理整个网页。为了解决这个瓶颈，我们设计了一种受检索启发的模型，通过对相关元素进行排序来高效修剪 HTML 页面。我们使用所选元素、屏幕截图和操作历史来评估各种模型在导航网页时模拟人类行为的能力。我们的实验从纯文本到专有的多模式 LLM 都有涉及。我们发现，较小的微调解码器超过了最好的零 - shot LLMs（包括 GPT-4V），但也超过了明确在屏幕截图上进行预训练的较大微调多模式模型。然而，所有微调模型都难以泛化到未见过的网站。我们的发现强调了需要能够泛化到新颖设置的大型多模式模型。