Large language models equipped with retrieval-augmented generation (RAG)
represent a burgeoning field aimed at enhancing answering capabilities by
leveraging external knowledge bases. Although the application of RAG with
language-only models has been extensively explored, its adaptation into
multimodal vision-language models remains nascent. Going beyond mere answer
generation, the primary goal of multimodal RAG is to cultivate the models'
ability to reason in response to relevant queries. To this end, we introduce a
novel multimodal RAG framework named RMR (Retrieval Meets Reasoning). The RMR
framework employs a bi-modal retrieval module to identify the most relevant
question-answer pairs, which then serve as scaffolds for the multimodal
reasoning process. This training-free approach not only encourages the model to
engage deeply with the reasoning processes inherent in the retrieved content
but also facilitates the generation of answers that are precise and richly
interpretable. Surprisingly, utilizing solely the ScienceQA dataset, collected
from elementary and high school science curricula, RMR significantly boosts the
performance of various vision-language models across a spectrum of benchmark
datasets, including A-OKVQA, MMBench, and SEED. These outcomes highlight the
substantial potential of our multimodal retrieval and reasoning mechanism to
improve the reasoning capabilities of vision-language models.

通过引入一种名为 RMR 的新型多模式 RAG 框架，本研究在多模式视觉语言模型中整合了基于检索的答案生成和推理能力，并通过提供相关问答对进行训练来显著提高各种基准数据集上的性能，凸显该框架在改善视觉语言模型的推理能力方面潜力巨大。