Multimodal reasoning is a critical component in the pursuit of artificial
intelligence systems that exhibit human-like intelligence, especially when
tackling complex tasks. While the chain-of-thought (CoT) technique has gained
considerable attention, the existing ScienceQA dataset, which focuses on
multimodal scientific questions and explanations from elementary and high
school textbooks, lacks a comprehensive evaluation of diverse approaches. To
address this gap, we present COCO Multi-Modal Reasoning Dataset(COCO-MMRD), a
novel dataset that encompasses an extensive collection of open-ended questions,
rationales, and answers derived from the large object dataset COCO. Unlike
previous datasets that rely on multiple-choice questions, our dataset pioneers
the use of open-ended questions in the context of multimodal CoT, introducing a
more challenging problem that effectively assesses the reasoning capability of
CoT models. Through comprehensive evaluations and detailed analyses, we provide
valuable insights and propose innovative techniques, including multi-hop
cross-modal attention and sentence-level contrastive learning, to enhance the
image and text encoders. Extensive experiments demonstrate the efficacy of the
proposed dataset and techniques, offering novel perspectives for advancing
multimodal reasoning.

多模态推理是实现类似人类智能的人工智能系统的关键组成部分，本文介绍了多模态链式思维（CoT）技术，并提出 COACO-MMRD 数据集，通过全面评估不同方法，提供了有价值的见解和创新技术，包括多跳跨模态注意力和句级对比学习，以增强图像和文本编码器的性能，通过广泛的实验证明了所提出数据集和技术的有效性，为推进多模态推理提供了新的视角。

增强类人多模态推理：一项新的具有挑战性的数据集和全面框架

Enhancing Human-like Multi-Modal Reasoning: A New Challenging Dataset  and Comprehensive Framework

We study the new problem of automatic question generation (QG) from
multi-modal sources containing images and texts, significantly expanding the
scope of most of the existing work that focuses exclusively on QG from only
textual sources. We propose a simple solution for our new problem, called
MultiQG-TI, which enables a text-only question generator to process visual
input in addition to textual input. Specifically, we leverage an image-to-text
model and an optical character recognition model to obtain the textual
description of the image and extract any texts in the image, respectively, and
then feed them together with the input texts to the question generator. We only
fine-tune the question generator while keeping the other components fixed. On
the challenging ScienceQA dataset, we demonstrate that MultiQG-TI significantly
outperforms ChatGPT with few-shot prompting, despite having hundred-times less
trainable parameters. Additional analyses empirically confirm the necessity of
both visual and textual signals for QG and show the impact of various modeling
choices.

我们研究了从多模态源（包含图像和文本）中自动生成问题（QG）的新问题，明显扩展了现有工作的范围，后者仅关注从文本源生成的 QG。我们提出了一个简单的解决方案，名为 MultiQG-TI，它使得仅基于文本的问题生成器能够处理视觉输入。我们通过利用图像到文本模型和光学字符识别模型来获得图像的文本描述并提取图像中的任何文本，然后将它们与输入文本一起馈送给问题生成器。在具有挑战性的 ScienceQA 数据集上，我们证明了 MultiQG-TI 在几次提示下明显优于 ChatGPT，尽管 MultiQG-TI 的可训练参数数量是 ChatGPT 的百倍少。额外的分析实验证实了 QG 所需的视觉和文本信号的必要性，并展示了各种建模选择的影响。