Autonomous agents that accomplish complex computer tasks with minimal human
interventions have the potential to transform human-computer interaction,
significantly enhancing accessibility and productivity. However, existing
benchmarks either lack an interactive environment or are limited to
environments specific to certain applications or domains, failing to reflect
the diverse and complex nature of real-world computer use, thereby limiting the
scope of tasks and agent scalability. To address this issue, we introduce
OSWorld, the first-of-its-kind scalable, real computer environment for
multimodal agents, supporting task setup, execution-based evaluation, and
interactive learning across various operating systems such as Ubuntu, Windows,
and macOS. OSWorld can serve as a unified, integrated computer environment for
assessing open-ended computer tasks that involve arbitrary applications.
Building upon OSWorld, we create a benchmark of 369 computer tasks involving
real web and desktop apps in open domains, OS file I/O, and workflows spanning
multiple applications. Each task example is derived from real-world computer
use cases and includes a detailed initial state setup configuration and a
custom execution-based evaluation script for reliable, reproducible evaluation.
Extensive evaluation of state-of-the-art LLM/VLM-based agents on OSWorld
reveals significant deficiencies in their ability to serve as computer
assistants. While humans can accomplish over 72.36% of the tasks, the best
model achieves only 12.24% success, primarily struggling with GUI grounding and
operational knowledge. Comprehensive analysis using OSWorld provides valuable
insights for developing multimodal generalist agents that were not possible
with previous benchmarks. Our code, environment, baseline models, and data are
publicly available at this https URL

通过引入 OSWorld，我们创建了一个包含 369 个计算机任务的基准，以评估多模态代理在开放领域中执行任意应用程序所涉及的计算机任务的能力。在 OSWorld 上进行的全面评估为开发多模态通用代理提供了宝贵的洞见，这是以前的基准测试无法实现的。

OSWorld: 多模态代理在真实计算机环境中的开放式任务评测

OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real  Computer Environments

Agents capable of carrying out general tasks on a computer can improve
efficiency and productivity by automating repetitive tasks and assisting in
complex problem-solving. Ideally, such agents should be able to solve new
computer tasks presented to them through natural language commands. However,
previous approaches to this problem require large amounts of expert
demonstrations and task-specific reward functions, both of which are
impractical for new tasks. In this work, we show that a pre-trained large
language model (LLM) agent can execute computer tasks guided by natural
language using a simple prompting scheme where the agent recursively criticizes
and improves its output (RCI). The RCI approach significantly outperforms
existing LLM methods for automating computer tasks and surpasses supervised
learning (SL) and reinforcement learning (RL) approaches on the MiniWoB++
benchmark. RCI is competitive with the state-of-the-art SL+RL method, using
only a handful of demonstrations per task rather than tens of thousands, and
without a task-specific reward function. Furthermore, we demonstrate RCI
prompting's effectiveness in enhancing LLMs' reasoning abilities on a suite of
natural language reasoning tasks, outperforming chain of thought (CoT)
prompting. We find that RCI combined with CoT performs better than either
separately.

本研究展示了一种使用 RCI 方法来自然语言执行计算机任务的代理方法，此方法能够显著提高计算机任务的自动化表现，优于现有的自然语言处理方法，并在自然语言推理任务中表现出较好的推理能力。