Large language models (LLMs) have achieved superior performance in powering
text-based AI agents, endowing them with decision-making and reasoning
abilities akin to humans. Concurrently, there is an emerging research trend
focused on extending these LLM-powered AI agents into the multimodal domain.
This extension enables AI agents to interpret and respond to diverse multimodal
user queries, thereby handling more intricate and nuanced tasks. In this paper,
we conduct a systematic review of LLM-driven multimodal agents, which we refer
to as large multimodal agents ( LMAs for short). First, we introduce the
essential components involved in developing LMAs and categorize the current
body of research into four distinct types. Subsequently, we review the
collaborative frameworks integrating multiple LMAs , enhancing collective
efficacy. One of the critical challenges in this field is the diverse
evaluation methods used across existing studies, hindering effective comparison
among different LMAs . Therefore, we compile these evaluation methodologies and
establish a comprehensive framework to bridge the gaps. This framework aims to
standardize evaluations, facilitating more meaningful comparisons. Concluding
our review, we highlight the extensive applications of LMAs and propose
possible future research directions. Our discussion aims to provide valuable
insights and guidelines for future research in this rapidly evolving field. An
up-to-date resource list is available at
this https URL

大型语言模型在多媒体领域的研究和发展方向的全面回顾，介绍了大型多媒体代理的开发要素和分类，评估方法的编制以及未来研究方向的提出。

大型多模态代理：综述

Large Multimodal Agents: A Survey

Language model agents (LMA) recently emerged as a promising paradigm on
muti-step decision making tasks, often outperforming humans and other
reinforcement learning agents. Despite the promise, their performance on
real-world applications that often involve combinations of tasks is still
underexplored. In this work, we introduce a new benchmark, called CompWoB -- 50
new compositional web automation tasks reflecting more realistic assumptions.
We show that while existing prompted LMAs (gpt-3.5-turbo or gpt-4) achieve
94.0% average success rate on base tasks, their performance degrades to 24.9%
success rate on compositional tasks. On the other hand, transferred LMAs
(finetuned only on base tasks) show less generalization gap, dropping from
85.4% to 54.8%. By balancing data distribution across tasks, we train a new
model, HTML-T5++, that surpasses human-level performance (95.2%) on MiniWoB,
and achieves the best zero-shot performance on CompWoB (61.5%). While these
highlight the promise of small-scale finetuned and transferred models for
compositional generalization, their performance further degrades under
different instruction compositions changing combinational order. In contrast to
the recent remarkable success of LMA, our benchmark and detailed analysis
emphasize the necessity of building LMAs that are robust and generalizable to
task compositionality for real-world deployment.

LMA 在基本任务上的表现优于人类和强化学习代理，但在涉及任务组合的实际应用中，它们的性能仍未得到充分探索。通过引入新的基准 CompWoB，我们展示了从基本任务到组合任务的 LMA 表现下降，而在训练过程中平衡任务数据分布后，我们设计了一个新模型 HTML-T5++，在 MiniWoB 上超过人类水平并在 CompWoB 上取得了最佳零样本性能。然而，在改变组合顺序的不同指令组合下，它们的性能进一步下降。与 LMA 的最新成功相反，我们的基准和详细分析突出了构建对真实世界部署具有稳健性和可泛化性的 LMA 的必要性。