Autonomous interaction with the computer has been a longstanding challenge
with great potential, and the recent proliferation of large language models
(LLMs) has markedly accelerated progress in building digital agents. However,
most of these agents are designed to interact with a narrow domain, such as a
specific software or website. This narrow focus constrains their applicability
for general computer tasks. To this end, we introduce OS-Copilot, a framework
to build generalist agents capable of interfacing with comprehensive elements
in an operating system (OS), including the web, code terminals, files,
multimedia, and various third-party applications. We use OS-Copilot to create
FRIDAY, a self-improving embodied agent for automating general computer tasks.
On GAIA, a general AI assistants benchmark, FRIDAY outperforms previous methods
by 35%, showcasing strong generalization to unseen applications via accumulated
skills from previous tasks. We also present numerical and quantitative evidence
that FRIDAY learns to control and self-improve on Excel and Powerpoint with
minimal supervision. Our OS-Copilot framework and empirical findings provide
infrastructure and insights for future research toward more capable and
general-purpose computer agents.

介绍了 OS-Copilot 框架，用于构建能够与操作系统中包括网络、代码终端、文件、多媒体和各种第三方应用程序在内的综合元素进行交互的通用型代理，并使用该框架创建了自我完善的具体化代理 FRIDAY，以自动化进行一般的计算机任务，在通用人工智能助理基准测试 GAIA 上，FRIDAY 通过积累来自以前任务的技能，表现出对未见过应用程序的强大泛化能力，超越了以前的方法 35％，同时提供了对 Excel 和 Powerpoint 进行控制和自我改进的数量和定量证据，该研究为未来更高能力和通用性的计算机代理提供了基础和见解。

OS-Copilot：面向具有自我改进能力的通才计算机代理

OS-Copilot: Towards Generalist Computer Agents with Self-Improvement

A kitchen assistant needs to operate human-scale objects, such as cabinets
and ovens, in unmapped environments with dynamic obstacles. Autonomous
interactions in such environments require integrating dexterous manipulation
and fluid mobility. While mobile manipulators in different form factors provide
an extended workspace, their real-world adoption has been limited. Executing a
high-level task for general objects requires a perceptual understanding of the
object as well as adaptive whole-body control among dynamic obstacles. In this
paper, we propose a two-stage architecture for autonomous interaction with
large articulated objects in unknown environments. The first stage,
object-centric planner, only focuses on the object to provide an
action-conditional sequence of states for manipulation using RGB-D data. The
second stage, agent-centric planner, formulates the whole-body motion control
as an optimal control problem that ensures safe tracking of the generated plan,
even in scenes with moving obstacles. We show that the proposed pipeline can
handle complex static and dynamic kitchen settings for both wheel-based and
legged mobile manipulators. Compared to other agent-centric planners, our
proposed planner achieves a higher success rate and a lower execution time. We
also perform hardware tests on a legged mobile manipulator to interact with
various articulated objects in a kitchen. For additional material, please
check: www.pair.toronto.edu/articulated-mm/.

本文提出了一个在未知环境中自主交互大型铰接物体的两阶段架构，第一阶段是以物体为中心的规划器，第二阶段是以智能代理为中心的规划器，并且在移动障碍物的情况下，通过优化控制问题确保安全跟踪所生成的计划。