Analogical reasoning is fundamental to human cognition and holds an important
place in various fields. However, previous studies mainly focus on single-modal
analogical reasoning and ignore taking advantage of structure knowledge.
Notably, the research in cognitive psychology has demonstrated that information
from multimodal sources always brings more powerful cognitive transfer than
single modality sources. To this end, we introduce the new task of multimodal
analogical reasoning over knowledge graphs, which requires multimodal reasoning
ability with the help of background knowledge. Specifically, we construct a
Multimodal Analogical Reasoning dataSet (MARS) and a multimodal knowledge graph
MarKG. We evaluate with multimodal knowledge graph embedding and pre-trained
Transformer baselines, illustrating the potential challenges of the proposed
task. We further propose a novel model-agnostic Multimodal analogical reasoning
framework with Transformer (MarT) motivated by the structure mapping theory,
which can obtain better performance. Code and datasets are available in
this https URL.

本研究介绍了基于知识图谱的多模式类比推理任务，并通过构建 Multimodal Analogical Reasoning 数据集和 Multimodal knowledge graph MarKG 进行评估。研究表明，从多模态源获取信息可以带来比单模态源更强大的认知转移，该研究进一步提出了一种基于结构映射理论的新型通用 Transformer 框架（MarT），能够取得更好的性能。