Inspired by the emergence of Large Language Models (LLMs) that can truly
understand human language, significant progress has been made in aligning
other, non-language, modalities to be `understandable' by an LLM, primarily via
converting their samples into a sequence of embedded language-like tokens
directly fed into the LLM (decoder) input stream. However, so far limited
attention has been given to transferring (and evaluating) one of the core LLM
capabilities to the emerging VLMs, namely the In-Context Learning (ICL)
ability, or in other words to guide VLMs to desired target downstream tasks or
output structure using in-context image+text demonstrations. In this work, we
dive deeper into analyzing the capabilities of some of the state-of-the-art
VLMs to follow ICL instructions, discovering them to be somewhat lacking. We
discover that even models that underwent large-scale mixed modality
pre-training and were implicitly guided to make use of interleaved image and
text information (intended to consume helpful context from multiple images)
under-perform when prompted with few-shot (ICL) demonstrations, likely due to
their lack of `direct' ICL instruction tuning. To test this conjecture, we
propose a simple, yet surprisingly effective, strategy of extending a common
VLM alignment framework with ICL support, methodology, and curriculum. We
explore, analyze, and provide insights into effective data mixes, leading up to
a significant 21.03% (and 11.3% on average) ICL performance boost over the
strongest VLM baselines and a variety of ICL benchmarks. We also contribute new
benchmarks for ICL evaluation in VLMs and discuss their advantages over the
prior art.

通过对一些最先进的 VLM（Visual Language Model）进行分析，我们发现它们在执行 ICL（In-Context Learning）指令时存在一定的不足。为了验证这个猜想，我们提出了一种简单但令人惊讶地有效的策略，通过扩展一个常见的 VLM 对齐框架，实现 ICL 支持、方法和课程设置。我们探讨、分析并提供了对有效数据混合的见解，从而显著提升了 21.03% 的 ICL 表现（平均 11.3%），超过了最强 VLM 基线和多种 ICL 基准，并为 VLM 的 ICL 评估贡献了新的基准，并讨论了它们相对于现有技术的优势。