Large Language Models (LLMs) demonstrate remarkable capabilities in question answering (QA), but metrics for assessing their reliance on memorization versus retrieval remain underdeveloped. Moreover, while finetuned models are state-of-the-art on closed-domain tasks, general-purpose models like GPT-4o exhibit strong zero-shot performance. This raises questions about the trade-offs between memorization, generalization, and retrieval. In this work, we analyze the extent to which multimodal retrieval-augmented VLMs memorize training data compared to baseline VLMs. Using the WebQA benchmark, we contrast finetuned models with baseline VLMs on multihop retrieval and question answering, examining the impact of finetuning on data memorization. To quantify memorization in end-to-end retrieval and QA systems, we propose several proxy metrics by investigating instances where QA succeeds despite retrieval failing. Our results reveal the extent to which finetuned models rely on memorization. In contrast, retrieval-augmented VLMs have lower memorization scores, at the cost of accuracy (72% vs 52% on WebQA test set). As such, our measures pose a challenge for future work to reconcile memorization and generalization in both Open-Domain QA and joint Retrieval-QA tasks.

本研究解决了大型语言模型在问答中对记忆与检索依赖程度评估不充分的问题。我们提出了几种代理指标，分析了多模态增强检索的视觉-语言模型在记忆训练数据方面的表现，并得出了经过微调的模型在记忆依赖性更强而准确性较低的关键发现。研究的潜在影响在于为未来在开放领域问答和联合检索-问答任务中调和记忆与泛化提供了新思路。

量化增强检索的视觉-语言模型中的记忆与检索性能