Large language models (LLMs) have shown impressive performance on complex
reasoning by leveraging chain-of-thought (CoT) prompting to generate
intermediate reasoning chains as the rationale to infer the answer. However,
existing CoT studies have focused on the language modality. We propose
Multimodal-CoT that incorporates language (text) and vision (images) modalities
into a two-stage framework that separates rationale generation and answer
inference. In this way, answer inference can leverage better generated
rationales that are based on multimodal information. With Multimodal-CoT, our
model under 1 billion parameters outperforms the previous state-of-the-art LLM
(GPT-3.5) by 16 percentage points (75.17%->91.68% accuracy) on the ScienceQA
benchmark and even surpasses human performance. Code is publicly available
available at this https URL.

该研究提出了一种将语言与视觉信息相结合的理由生成和答案推断的多模态 - CoT 框架，使得答案推断可以更好地利用基于多模态信息的生成的理由，并取得了比先前最先进的 LLM（GPT-3.5）高 16 个百分点（75.17％-> 91.68％准确度）的性能，在 ScienceQA 基准测试中甚至超过了人类的表现。