The reasoning capabilities of LLM (Large Language Model) are widely
acknowledged in recent research, inspiring studies on tool learning and
autonomous agents. LLM serves as the "brain" of agent, orchestrating multiple
tools for collaborative multi-step task solving. Unlike methods invoking tools
like calculators or weather APIs for straightforward tasks, multi-modal agents
excel by integrating diverse AI models for complex challenges. However, current
multi-modal agents neglect the significance of model selection: they primarily
focus on the planning and execution phases, and will only invoke predefined
task-specific models for each subtask, making the execution fragile. Meanwhile,
other traditional model selection methods are either incompatible with or
suboptimal for the multi-modal agent scenarios, due to ignorance of
dependencies among subtasks arising by multi-step reasoning.
To this end, we identify the key challenges therein and propose the
$\textit{M}^3$ framework as a plug-in with negligible runtime overhead at
test-time. This framework improves model selection and bolsters the robustness
of multi-modal agents in multi-step reasoning. In the absence of suitable
benchmarks, we create MS-GQA, a new dataset specifically designed to
investigate the model selection challenge in multi-modal agents. Our
experiments reveal that our framework enables dynamic model selection,
considering both user inputs and subtask dependencies, thereby robustifying the
overall reasoning process. Our code and benchmark:
this https URL

我们提出了一个名为 $	extit {M}^3$ 框架的插件，用于提高多模态代理在多步推理中的模型选择和鲁棒性，同时创建了一个新的数据集 MS-GQA，用于研究多模态代理中的模型选择挑战，并通过实验证明我们的框架能够动态地选择模型，考虑用户输入和子任务之间的依赖关系，从而增强整体推理过程的鲁棒性。