Recent advancements in large foundation models have remarkably enhanced our
understanding of sensory information in open-world environments. In leveraging
the power of foundation models, it is crucial for AI research to pivot away
from excessive reductionism and toward an emphasis on systems that function as
cohesive wholes. Specifically, we emphasize developing Agent AI -- an embodied
system that integrates large foundation models into agent actions. The emerging
field of Agent AI spans a wide range of existing embodied and agent-based
multimodal interactions, including robotics, gaming, and healthcare systems,
etc. In this paper, we propose a novel large action model to achieve embodied
intelligent behavior, the Agent Foundation Model. On top of this idea, we
discuss how agent AI exhibits remarkable capabilities across a variety of
domains and tasks, challenging our understanding of learning and cognition.
Furthermore, we discuss the potential of Agent AI from an interdisciplinary
perspective, underscoring AI cognition and consciousness within scientific
discourse. We believe that those discussions serve as a basis for future
research directions and encourage broader societal engagement.

利用大型基础模型提高我们对开放环境中感知信息的理解，在 AI 研究中强调发展集成大型基础模型的代理人 AI，这一新兴领域涉及机器人技术、游戏和医疗等多种具身和基于代理的多模态交互，并讨论了代理人 AI 在各个领域和任务中展现出的能力，挑战我们对学习和认知的理解，通过跨学科视角探讨了代理人 AI 的潜力，并强调将其纳入科学讨论中，为未来的研究方向提供基础并促进更广泛的社会参与。

位置论文：走向整体智能的智能代理

Position Paper: Agent AI Towards a Holistic Intelligence

Multi-modal AI systems will likely become a ubiquitous presence in our
everyday lives. A promising approach to making these systems more interactive
is to embody them as agents within physical and virtual environments. At
present, systems leverage existing foundation models as the basic building
blocks for the creation of embodied agents. Embedding agents within such
environments facilitates the ability of models to process and interpret visual
and contextual data, which is critical for the creation of more sophisticated
and context-aware AI systems. For example, a system that can perceive user
actions, human behavior, environmental objects, audio expressions, and the
collective sentiment of a scene can be used to inform and direct agent
responses within the given environment. To accelerate research on agent-based
multimodal intelligence, we define "Agent AI" as a class of interactive systems
that can perceive visual stimuli, language inputs, and other
environmentally-grounded data, and can produce meaningful embodied action with
infinite agent. In particular, we explore systems that aim to improve agents
based on next-embodied action prediction by incorporating external knowledge,
multi-sensory inputs, and human feedback. We argue that by developing agentic
AI systems in grounded environments, one can also mitigate the hallucinations
of large foundation models and their tendency to generate environmentally
incorrect outputs. The emerging field of Agent AI subsumes the broader embodied
and agentic aspects of multimodal interactions. Beyond agents acting and
interacting in the physical world, we envision a future where people can easily
create any virtual reality or simulated scene and interact with agents embodied
within the virtual environment.

多模态人工智能系统中，基于 Agent 的多模态智能研究，通过在真实和虚拟环境中嵌入具有感知能力的智能体，能够处理和解释视觉和语境数据，以提高智能系统的上下文感知和交互能力。