Multi-modal and multi-hop question answering aims to answer a question based
on multiple input sources from different modalities. Previous methods retrieve
the evidence separately and feed the retrieved evidence to a language model to
generate the corresponding answer. However, these methods fail to build
connections between candidates and thus cannot model the inter-dependent
relation during retrieval. Moreover, the reasoning process over multi-modality
candidates can be unbalanced without building alignments between different
modalities. To address this limitation, we propose a Structured Knowledge and
Unified Retrieval Generation based method (SKURG). We align the sources from
different modalities via the shared entities and map them into a shared
semantic space via structured knowledge. Then, we utilize a unified
retrieval-generation decoder to integrate intermediate retrieval results for
answer generation and adaptively determine the number of retrieval steps. We
perform experiments on two multi-modal and multi-hop datasets: WebQA and
MultimodalQA. The results demonstrate that SKURG achieves state-of-the-art
performance on both retrieval and answer generation.

提出了一种基于结构化知识和统一检索生成的方法（SKURG）来回答多模态和多跳问题，包括模态对齐，统一检索生成和中间检索结果的集成。该方法在两个数据集上均取得了最先进的效果。