Recent advancements in Multi-modal Large Language Models (MLLMs) have
significantly improved their performance in tasks combining vision and
language. However, challenges persist in detailed multi-modal understanding,
comprehension of complex tasks, and reasoning over multi-modal information.
This paper introduces MMCTAgent, a novel multi-modal critical thinking agent
framework designed to address the inherent limitations of current MLLMs in
complex visual reasoning tasks. Inspired by human cognitive processes and
critical thinking, MMCTAgent iteratively analyzes multi-modal information,
decomposes queries, plans strategies, and dynamically evolves its reasoning.
Additionally, MMCTAgent incorporates critical thinking elements such as
verification of final answers and self-reflection through a novel approach that
defines a vision-based critic and identifies task-specific evaluation criteria,
thereby enhancing its decision-making abilities. Through rigorous evaluations
across various image and video understanding benchmarks, we demonstrate that
MMCTAgent (with and without the critic) outperforms both foundational MLLMs and
other tool-augmented pipelines.

这篇论文介绍了 MMCTAgent，这是一个新颖的多模态批判性思维代理框架，旨在解决当前 MLLM 在复杂视觉推理任务中的固有限制。通过从人类认知过程和批判性思维中获得灵感，MMCTAgent 迭代地分析多模态信息，分解查询，计划策略，并动态发展其推理能力。此外，MMCTAgent 还融入了验证最终答案和自我反思的批判性思维元素，通过定义基于视觉的评论员并识别任务特定的评估标准，从而增强其决策能力。通过在各种图像和视频理解基准上进行严格评估，我们证明 MMCTAgent（有和无评论员）优于基础 MLLM 和其他工具增强的流水线。

MMCTAgent：复杂视觉推理的多模态批判性思维代理框架

MMCTAgent: Multi-modal Critical Thinking Agent Framework for Complex  Visual Reasoning

In recent times there has been a surge of multi-modal architectures based on
Large Language Models, which leverage the zero shot generation capabilities of
LLMs and project image embeddings into the text space and then use the
auto-regressive capacity to solve tasks such as VQA, captioning, and image
retrieval. We name these architectures as "bridge-architectures" as they
project from the image space to the text space. These models deviate from the
traditional recipe of training transformer based multi-modal models, which
involve using large-scale pre-training and complex multi-modal interactions
through co or cross attention. However, the capabilities of bridge
architectures have not been tested on complex visual reasoning tasks which
require fine grained analysis about the image. In this project, we investigate
the performance of these bridge-architectures on the NLVR2 dataset, and compare
it to state-of-the-art transformer based architectures. We first extend the
traditional bridge architectures for the NLVR2 dataset, by adding object level
features to faciliate fine-grained object reasoning. Our analysis shows that
adding object level features to bridge architectures does not help, and that
pre-training on multi-modal data is key for good performance on complex
reasoning tasks such as NLVR2. We also demonstrate some initial results on a
recently bridge-architecture, LLaVA, in the zero shot setting and analyze its
performance.

探讨了多模态架构在复杂视觉推理任务上的表现，发现在 NLVR2 数据集中，使用对象级别特征的桥梁架构并不能提高性能，而预训练多模态数据对于复杂推理任务的良好表现是关键。还对最新的桥梁架构 LLaVA 在零 - shot 环境中的性能进行了初步分析。