Multimodal reasoning with large language models (LLMs) often suffers from
hallucinations and the presence of deficient or outdated knowledge within LLMs.
Some approaches have sought to mitigate these issues by employing textual
knowledge graphs, but their singular modality of knowledge limits comprehensive
cross-modal understanding. In this paper, we propose the Multimodal Reasoning
with Multimodal Knowledge Graph (MR-MKG) method, which leverages multimodal
knowledge graphs (MMKGs) to learn rich and semantic knowledge across
modalities, significantly enhancing the multimodal reasoning capabilities of
LLMs. In particular, a relation graph attention network is utilized for
encoding MMKGs and a cross-modal alignment module is designed for optimizing
image-text alignment. A MMKG-grounded dataset is constructed to equip LLMs with
initial expertise in multimodal reasoning through pretraining. Remarkably,
MR-MKG achieves superior performance while training on only a small fraction of
parameters, approximately 2.25% of the LLM's parameter size. Experimental
results on multimodal question answering and multimodal analogy reasoning tasks
demonstrate that our MR-MKG method outperforms previous state-of-the-art
models.

提出了一种利用多模态知识图 (MMKGs) 进行多模态推理的方法 (MR-MKG)，通过利用多模态知识图 (MMKGs) 跨模态地学习丰富的语义知识，显著提高了大型语言模型 (LLMs) 在多模态推理中的能力。通过在仅使用 LLM 参数的 2.25% 进行训练，MR-MKG 在多模态问答和多模态类比推理任务上取得了优越的性能，胜过了之前的最新模型。