Large Multimodal Models (LMMs) excel in natural language and visual
understanding but are challenged by exacting tasks such as Knowledge-based
Visual Question Answering (KB-VQA) which involve the retrieval of relevant
information from document collections to use in shaping answers to questions.
We present an extensive training and evaluation framework, M2KR, for KB-VQA.
M2KR contains a collection of vision and language tasks which we have
incorporated into a single suite of benchmark tasks for training and evaluating
general-purpose multi-modal retrievers. We use M2KR to develop PreFLMR, a
pre-trained version of the recently developed Fine-grained Late-interaction
Multi-modal Retriever (FLMR) approach to KB-VQA, and we report new
state-of-the-art results across a range of tasks. We also present
investigations into the scaling behaviors of PreFLMR intended to be useful in
future developments in general-purpose multi-modal retrievers.

用于知识驱动的视觉问答的大规模多模型训练和评估框架 M2KR，通过发展 PreFLMR 模型，在一系列任务中取得了最新的最先进的结果，并研究了 PreFLMR 的扩展性行为，为未来的多模型检索系统开发提供了有用的参考。