We study the use of large language model-based agents for interacting with
software via web browsers. Unlike prior work, we focus on measuring the agents'
ability to perform tasks that span the typical daily work of knowledge workers
utilizing enterprise software systems. To this end, we propose WorkArena, a
remote-hosted benchmark of 29 tasks based on the widely-used ServiceNow
platform. We also introduce BrowserGym, an environment for the design and
evaluation of such agents, offering a rich set of actions as well as multimodal
observations. Our empirical evaluation reveals that while current agents show
promise on WorkArena, there remains a considerable gap towards achieving full
task automation. Notably, our analysis uncovers a significant performance
disparity between open and closed-source LLMs, highlighting a critical area for
future exploration and development in the field.

使用基于大型语言模型的代理程序研究通过 web 浏览器与软件的交互。通过提出基于 ServiceNow 平台的 29 个任务的远程托管基准 WorkArena，以及设计和评估此类代理程序的环境 BrowserGym，我们对其性能进行了实证评估，发现目前代理程序在工作区域上有一些潜力，但在实现完全的任务自动化方面还存在相当大的差距。尤其值得注意的是，我们的分析揭示了开源和闭源 LLM（大型语言模型）之间的显著性能差异，突出了该领域未来研究和发展的重要方向。