Multimodal reasoning is a critical component in the pursuit of artificial
intelligence systems that exhibit human-like intelligence, especially when
tackling complex tasks. While the chain-of-thought (CoT) technique has gained
considerable attention, the existing ScienceQA dataset, which focuses on
multimodal scientific questions and explanations from elementary and high
school textbooks, lacks a comprehensive evaluation of diverse approaches. To
address this gap, we present COCO Multi-Modal Reasoning Dataset(COCO-MMRD), a
novel dataset that encompasses an extensive collection of open-ended questions,
rationales, and answers derived from the large object dataset COCO. Unlike
previous datasets that rely on multiple-choice questions, our dataset pioneers
the use of open-ended questions in the context of multimodal CoT, introducing a
more challenging problem that effectively assesses the reasoning capability of
CoT models. Through comprehensive evaluations and detailed analyses, we provide
valuable insights and propose innovative techniques, including multi-hop
cross-modal attention and sentence-level contrastive learning, to enhance the
image and text encoders. Extensive experiments demonstrate the efficacy of the
proposed dataset and techniques, offering novel perspectives for advancing
multimodal reasoning.

多模态推理是实现类似人类智能的人工智能系统的关键组成部分，本文介绍了多模态链式思维（CoT）技术，并提出 COACO-MMRD 数据集，通过全面评估不同方法，提供了有价值的见解和创新技术，包括多跳跨模态注意力和句级对比学习，以增强图像和文本编码器的性能，通过广泛的实验证明了所提出数据集和技术的有效性，为推进多模态推理提供了新的视角。