In the real world, knowledge often exists in a multimodal and heterogeneous
form. Addressing the task of question answering with hybrid data types,
including text, tables, and images, is a challenging task (MMHQA). Recently,
with the rise of large language models (LLM), in-context learning (ICL) has
become the most popular way to solve QA problems. We propose MMHQA-ICL
framework for addressing this problems, which includes stronger heterogeneous
data retriever and an image caption module. Most importantly, we propose a
Type-specific In-context Learning Strategy for MMHQA, enabling LLMs to leverage
their powerful performance in this task. We are the first to use end-to-end LLM
prompting method for this task. Experimental results demonstrate that our
framework outperforms all baselines and methods trained on the full dataset,
achieving state-of-the-art results under the few-shot setting on the
MultimodalQA dataset.

多模态问题回答（MMHQA）是一个具有挑战性的任务，我们提出了一个 MMHQA-ICL 框架，包括强大的异构数据检索器和图像描述模块，并首次使用了端到端 LLM 提示方法。实验结果表明，我们的框架在多模态问题回答上表现出色，优于所有基准线和使用完整数据集训练的方法，达到了多模态 QA 数据集在少样本设置下的最新成果。