Vision-and-Language Navigation (VLN) task aims to enable AI agents to
accurately understand and follow natural language instructions to navigate
through real-world environments, ultimately reaching specific target locations.
We recognise a promising opportunity to extend VLN to a comparable navigation
task that holds substantial significance in our daily lives, albeit within the
virtual realm: navigating websites on the Internet. This paper proposes a new
task named Vision-and-Language Navigation on Websites (WebVLN), where we use
question-based instructions to train an agent, emulating how users naturally
browse websites. Unlike the existing VLN task that only pays attention to
vision and instruction (language), the WebVLN agent further considers
underlying web-specific content like HTML, which could not be seen on the
rendered web pages yet contains rich visual and textual information. Toward
this goal, we contribute a dataset, WebVLN-v1, and introduce a novel approach
called Website-aware VLN Network (WebVLN-Net), which is built upon the
foundation of state-of-the-art VLN techniques. Experimental results show that
WebVLN-Net outperforms current VLN and web-related navigation methods. We
believe that the introduction of the new WebVLN task and its dataset will
establish a new dimension within the VLN domain and contribute to the broader
vision-and-language research community. The code is available at:
this https URL

该研究提出了一项名为 WebVLN 的新任务，利用问答形式的指令训练代理程序，模拟用户在网页上浏览的过程。与现有的 Vision-and-Language Navigation 任务相比，WebVLN 代理程序进一步考虑 HTML 等网页特定内容，通过构建在最先进的 VLN 技术基础之上的 WebVLN-Net 方法，在 VLN 和网页导航方法上取得了更好的表现。该研究的贡献在于创建了 WebVLN-v1 数据集，并为 VLN 领域与广泛的视觉与语言研究社区做出了新的探索。