Multimodal Retrieval Augmented Generation (MMRAG) is a powerful approach to
question-answering over multimodal documents. A key challenge with evaluating
MMRAG is the paucity of high-quality datasets matching the question styles and
modalities of interest. In light of this, we propose SMMQG, a synthetic data
generation framework. SMMQG leverages interplay between a retriever, large
language model (LLM) and large multimodal model (LMM) to generate question and
answer pairs directly from multimodal documents, with the questions conforming
to specified styles and modalities. We use SMMQG to generate an MMRAG dataset
of 1024 questions over Wikipedia documents and evaluate state-of-the-art models
using it, revealing insights into model performance that are attainable only
through style- and modality-specific evaluation data. Next, we measure the
quality of data produced by SMMQG via a human study. We find that the quality
of our synthetic data is on par with the quality of the crowdsourced benchmark
MMQA and that downstream evaluation results using both datasets strongly
concur.

使用 SMMQG 生成合成数据，该方法结合了检索器、大型语言模型和大型多模型，可以直接从多模态文档中生成符合特定样式和形式的问题和答案对，用于对多模态文档进行问答评估，并与现有数据集 MMQA 进行质量对比。