A long-standing goal of AI systems is to perform complex multimodal reasoning
like humans. Recently, large language models (LLMs) have made remarkable
strides in such multi-step reasoning on the language modality solely by
leveraging the chain of thought (CoT) to mimic human thinking. However, the
transfer of these advancements to multimodal contexts introduces heightened
challenges, including but not limited to the impractical need for
labor-intensive annotation and the limitations in terms of flexibility,
generalizability, and explainability. To evoke CoT reasoning in multimodality,
this work first conducts an in-depth analysis of these challenges posed by
multimodality and presents two key insights: "keeping critical thinking" and
"letting everyone do their jobs" in multimodal CoT reasoning. Furthermore, this
study proposes a novel DDCoT prompting that maintains a critical attitude
through negative-space prompting and incorporates multimodality into reasoning
by first dividing the reasoning responsibility of LLMs into reasoning and
recognition and then integrating the visual recognition capability of visual
models into the joint reasoning process. The rationales generated by DDCoT not
only improve the reasoning abilities of both large and small language models in
zero-shot prompting and fine-tuning learning, significantly outperforming
state-of-the-art methods but also exhibit impressive generalizability and
explainability.

AI 系统的一个长期目标是像人类一样进行复杂的多模态推理。最近，大型语言模型（LLMs）通过利用思维链（CoT）在仅使用语言模态上取得了显著的多步推理进展，然而，将这些进展应用于多模态情境引入了更高的挑战，其中包括但不限于对劳动密集型注释的不切实际需求以及在灵活性、泛化性和可解释性方面的限制。为了在多模态中唤起 CoT 推理，该研究首先对多模态提出的这些挑战进行了深入分析，并提出了两个关键见解：“保持批判性思维” 和 “让每个人发挥各自的作用” 在多模态 CoT 推理中。此外，该研究提出了一种新颖的 DDCoT 提示，通过负空间提示保持临界态度，并通过首先将 LLMs 的推理责任划分为推理和识别，然后将视觉模型的视觉识别能力整合到联合推理过程中来融入多模态推理。DDCoT 生成的基于理性的解释不仅改进了大型和小型语言模型在零样本提示和微调学习中的推理能力，显著超过了最先进的方法，而且还展示出令人印象深刻的泛化性和可解释性。