Multi-modal AI systems will likely become a ubiquitous presence in our
everyday lives. A promising approach to making these systems more interactive
is to embody them as agents within physical and virtual environments. At
present, systems leverage existing foundation models as the basic building
blocks for the creation of embodied agents. Embedding agents within such
environments facilitates the ability of models to process and interpret visual
and contextual data, which is critical for the creation of more sophisticated
and context-aware AI systems. For example, a system that can perceive user
actions, human behavior, environmental objects, audio expressions, and the
collective sentiment of a scene can be used to inform and direct agent
responses within the given environment. To accelerate research on agent-based
multimodal intelligence, we define "Agent AI" as a class of interactive systems
that can perceive visual stimuli, language inputs, and other
environmentally-grounded data, and can produce meaningful embodied action with
infinite agent. In particular, we explore systems that aim to improve agents
based on next-embodied action prediction by incorporating external knowledge,
multi-sensory inputs, and human feedback. We argue that by developing agentic
AI systems in grounded environments, one can also mitigate the hallucinations
of large foundation models and their tendency to generate environmentally
incorrect outputs. The emerging field of Agent AI subsumes the broader embodied
and agentic aspects of multimodal interactions. Beyond agents acting and
interacting in the physical world, we envision a future where people can easily
create any virtual reality or simulated scene and interact with agents embodied
within the virtual environment.

多模态人工智能系统中，基于 Agent 的多模态智能研究，通过在真实和虚拟环境中嵌入具有感知能力的智能体，能够处理和解释视觉和语境数据，以提高智能系统的上下文感知和交互能力。

Agent AI：多模态交互视野调查

Agent AI: Surveying the Horizons of Multimodal Interaction

This paper surveys vision-language pre-training (VLP) methods for multimodal
intelligence that have been developed in the last few years. We group these
approaches into three categories: ($i$) VLP for image-text tasks, such as image
captioning, image-text retrieval, visual question answering, and visual
grounding; ($ii$) VLP for core computer vision tasks, such as (open-set) image
classification, object detection, and segmentation; and ($iii$) VLP for
video-text tasks, such as video captioning, video-text retrieval, and video
question answering. For each category, we present a comprehensive review of
state-of-the-art methods, and discuss the progress that has been made and
challenges still being faced, using specific systems and models as case
studies. In addition, for each category, we discuss advanced topics being
actively explored in the research community, such as big foundation models,
unified modeling, in-context few-shot learning, knowledge, robustness, and
computer vision in the wild, to name a few.

本文研究多模态智能领域的视觉 - 语言预训练方法，分为三类分类，包含图像 - 文本、核心计算机视觉和视频 - 文本任务，针对每类任务，提出了针对性的方法，分别探究了研究进展和存在的挑战并讨论了更先进的主题。

视觉 - 语言预训练：基础、最新进展和未来趋势

Vision-Language Pre-training: Basics, Recent Advances, and Future Trends

Deep learning methods have revolutionized speech recognition, image
recognition, and natural language processing since 2010. Each of these tasks
involves a single modality in their input signals. However, many applications
in the artificial intelligence field involve multiple modalities. Therefore, it
is of broad interest to study the more difficult and complex problem of
modeling and learning across multiple modalities. In this paper, we provide a
technical review of available models and learning methods for multimodal
intelligence. The main focus of this review is the combination of vision and
natural language modalities, which has become an important topic in both the
computer vision and natural language processing research communities. This
review provides a comprehensive analysis of recent works on multimodal deep
learning from three perspectives: learning multimodal representations, fusing
multimodal signals at various levels, and multimodal applications. Regarding
multimodal representation learning, we review the key concepts of embedding,
which unify multimodal signals into a single vector space and thereby enable
cross-modality signal processing. We also review the properties of many types
of embeddings that are constructed and learned for general downstream tasks.
Regarding multimodal fusion, this review focuses on special architectures for
the integration of representations of unimodal signals for a particular task.
Regarding applications, selected areas of a broad interest in the current
literature are covered, including image-to-text caption generation,
text-to-image generation, and visual question answering. We believe that this
review will facilitate future studies in the emerging field of multimodal
intelligence for related communities.

本文综述了多模态智能领域中的各种模型和学习方法。主要关注点是视觉和自然语言模态的组合，涵盖了多模态表示学习、多模态信号融合以及多模态应用等方面，旨在为相关社群未来的研究提供参考。