With the advent of large language models(LLMs) enhanced by the
chain-of-thought(CoT) methodology, visual reasoning problem is usually
decomposed into manageable sub-tasks and tackled sequentially with various
external tools. However, such a paradigm faces the challenge of the potential
"determining hallucinations" in decision-making due to insufficient visual
information and the limitation of low-level perception tools that fail to
provide abstract summaries necessary for comprehensive reasoning. We argue that
converging visual context acquisition and logical reasoning is pivotal for
tackling visual reasoning tasks. This paper delves into the realm of multimodal
CoT to solve intricate visual reasoning tasks with multimodal large language
models(MLLMs) and their cognitive capability. To this end, we propose an
innovative multimodal CoT framework, termed Cantor, characterized by a
perception-decision architecture. Cantor first acts as a decision generator and
integrates visual inputs to analyze the image and problem, ensuring a closer
alignment with the actual context. Furthermore, Cantor leverages the advanced
cognitive functions of MLLMs to perform as multifaceted experts for deriving
higher-level information, enhancing the CoT generation process. Our extensive
experiments demonstrate the efficacy of the proposed framework, showing
significant improvements in multimodal CoT performance across two complex
visual reasoning datasets, without necessitating fine-tuning or ground-truth
rationales. Project Page: this https URL .

通过结合视觉环境获取和逻辑推理，本论文提出了一种名为 Cantor 的创新的多模态 CoT 框架，利用多模态大语言模型的认知能力解决复杂的视觉推理任务，该框架通过分析图像和问题实现对视觉输入的整合，利用大语言模型的高级认知功能生成更高级别的信息，提高了多模态 CoT 性能。

Cantor: MLLM 的多模态思维链激发

Cantor: Inspiring Multimodal Chain-of-Thought of MLLM

Given that rich information is hidden behind ubiquitous numbers in text,
numerical reasoning over text should be an essential skill of AI systems. To
derive precise equations to solve numerical reasoning problems, previous work
focused on modeling the structures of equations, and has proposed various
structured decoders. Though structure modeling proves to be effective, these
structured decoders construct a single equation in a pre-defined autoregressive
order, potentially placing an unnecessary restriction on how a model should
grasp the reasoning process. Intuitively, humans may have numerous pieces of
thoughts popping up in no pre-defined order; thoughts are not limited to the
problem at hand, and can even be concerned with other related problems. By
comparing diverse thoughts and chaining relevant pieces, humans are less prone
to errors. In this paper, we take this inspiration and propose CANTOR, a
numerical reasoner that models reasoning steps using a directed acyclic graph
where we produce diverse reasoning steps simultaneously without pre-defined
decoding dependencies, and compare and chain relevant ones to reach a solution.
Extensive experiments demonstrated the effectiveness of CANTOR under both
fully-supervised and weakly-supervised settings.

本文提出了一种名为 CANTOR 的数值推理算法，在不预定义解码依赖关系的情况下同时生成多种推理步骤，并比较并链接相关步骤以达到求解问题的目的。