Despite impressive advances in recent multimodal large language models
(MLLMs), state-of-the-art models such as from the GPT-4 suite still struggle
with knowledge-intensive tasks. To address this, we consider Reverse Image
Retrieval (RIR) augmented generation, a simple yet effective strategy to
augment MLLMs with web-scale reverse image search results. RIR robustly
improves knowledge-intensive visual question answering (VQA) of GPT-4V by
37-43%, GPT-4 Turbo by 25-27%, and GPT-4o by 18-20% in terms of open-ended VQA
evaluation metrics. To our surprise, we discover that RIR helps the model to
better access its own world knowledge. Concretely, our experiments suggest that
RIR augmentation helps by providing further visual and textual cues without
necessarily containing the direct answer to a query. In addition, we elucidate
cases in which RIR can hurt performance and conduct a human evaluation.
Finally, we find that the overall advantage of using RIR makes it difficult for
an agent that can choose to use RIR to perform better than an approach where
RIR is the default setting.

使用 Reverse Image Retrieval（RIR）辅助生成的简单而有效的策略，对于 GPT-4 系列等最先进的多模态大型语言模型（MLLM）来说，RIR 显著提高了知识密集型的视觉问答性能，通过开放式 VQA 评估指标，GPT-4V 提高 37-43％，GPT-4 Turbo 提高 25-27％，GPT-4o 提高 18-20％。