We investigate the generalization boundaries of current Multimodal Large
Language Models (MLLMs) via comprehensive evaluation under out-of-distribution
scenarios and domain-specific tasks. We evaluate their zero-shot generalization
across synthetic images, real-world distributional shifts, and specialized
datasets like medical and molecular imagery. Empirical results indicate that
MLLMs struggle with generalization beyond common training domains, limiting
their direct application without adaptation. To understand the cause of
unreliable performance, we analyze three hypotheses: semantic
misinterpretation, visual feature extraction insufficiency, and mapping
deficiency. Results identify mapping deficiency as the primary hurdle. To
address this problem, we show that in-context learning (ICL) can significantly
enhance MLLMs' generalization, opening new avenues for overcoming
generalization barriers. We further explore the robustness of ICL under
distribution shifts and show its vulnerability to domain shifts, label shifts,
and spurious correlation shifts between in-context examples and test data.

通过广泛的评估工作，我们研究了当前多模态大型语言模型在超出分布情景和特定领域任务下的泛化边界。实证结果表明，多模态大型语言模型在常见训练域之外的泛化上存在困难，需要经过适应性的直接应用。我们进一步探索了上下文学习在分布偏移下的鲁棒性，并展示了它在领域偏移、标签偏移和上下文示例与测试数据之间的虚假相关偏移中的脆弱性。