Creating autonomous virtual agents capable of using arbitrary software on any
digital device remains a major challenge for artificial intelligence. Two key
obstacles hinder progress: insufficient infrastructure for building virtual
agents in real-world environments, and the need for in-the-wild evaluation of
fundamental agent abilities. To address this, we introduce AgentStudio, an
online, realistic, and multimodal toolkit that covers the entire lifecycle of
agent development. This includes environment setups, data collection, agent
evaluation, and visualization. The observation and action spaces are highly
generic, supporting both function calling and human-computer interfaces. This
versatility is further enhanced by AgentStudio's graphical user interfaces,
which allow efficient development of datasets and benchmarks in real-world
settings. To illustrate, we introduce a visual grounding dataset and a
real-world benchmark suite, both created with our graphical interfaces.
Furthermore, we present several actionable insights derived from AgentStudio,
e.g., general visual grounding, open-ended tool creation, learning from videos,
etc. We have open-sourced the environments, datasets, benchmarks, and
interfaces to promote research towards developing general virtual agents for
the future.

创建具备适应任意软件在任何数字设备上运行的自主虚拟代理人仍然是人工智能的一个重大挑战。为了解决这个问题，我们介绍了 AgentStudio，这是一个在线、逼真且多模态的工具包，涵盖了代理人开发的整个生命周期，包括环境设置、数据收集、代理人评估和可视化等。